DataSource rename to BackendTasks, and Exception Handling

[dillonkearns]

Background

With the new features in the elm-pages v3 beta, I've taken a step back to think about the core abstraction of elm-pages. For example, with server-rendered routes, and actions, DataSource's have expanded to do more than just pulling in static data. Now it is used for:

• Pulling in dynamic data at request-time
• Performing side-effects (like hitting an API or database)
• Run scripting steps with Scripts through the elm-pages run command

This means it is likely you will want to do more sophisticated things like recovering from HTTP errors (if an HTTP API request fails with validations, for example). And the abstraction isn't only about pulling in data - when the only goal is to fire and forget an effectful API request, the term DataSource doesn't fit so well.

Changes: Rename to BackendTask and Exceptions

To better reflect this, I'm making 3 changes around DataSources:

• Renaming DataSource -> BackendTask
• Adding an error type variable DataSource value -> BackendTask error value
• Adding an Exception type, which allows errors to be passed to the framework and give a CLI error code or a server error code (as appropriate for the context)
• You can tell whether a BackendTask can cause an error or not by its error type variable - for example, BackendTask.succeed "Hello" gives you BackendTask error String. You don't need to usethrow when there are no possible errors like in BackendTask.succeed

Here's what that looks like in practice. A few things to notice in this example:

• The Exception type is useful for taking a BackendTask's error values in one of two paths: throwing to ignore the error and pass it up to the framework if something goes wrong, or catch to unwrap the nicely typed error data so you can handle it
• BackendTask.throw : BackendTask (Catchable error) data -> BackendTask Throwable data - this is in contrast to BackendTask.catch : BackendTask (Catchable error) value -> BackendTask error value
• The core elm-pages APIs like BackendTask.File, BackendTask.Glob, BackendTask.Http have nicely typed error data and return their own error types wrapped with Catchable. But users can create Catchable error types to enable this same flow, or they can directly create a Throwable which can lets the framework handle the unexpected error

import BackendTask exposing (BackendTask)
import BackendTask.File
import BackendTask.Glob as Glob exposing (Glob)
import Exception exposing (Throwable)
import List.Extra
import Markdown.Block as Block exposing (Block)
import Markdown.Parser
import Markdown.Renderer


titleForSection : Section -> BackendTask Throwable NextPrevious.Item
titleForSection section =
    Glob.expectUniqueMatch (findBySlug section.slug)
        |> BackendTask.throw
        |> BackendTask.andThen
            (\filePath ->
                BackendTask.File.bodyWithoutFrontmatter filePath
                    |> BackendTask.throw
                    |> BackendTask.andThen markdownBody
                    |> BackendTask.map
                        (\blocks ->
                            List.Extra.findMap
                                (\block ->
                                    case block of
                                        Block.Heading Block.H1 inlines ->
                                            Just
                                                { title = Block.extractInlineText inlines
                                                , slug = section.slug
                                                }

                                        _ ->
                                            Nothing
                                )
                                blocks
                        )
            )
        |> BackendTask.andThen
            (\maybeTitle ->
                maybeTitle
                    |> Result.fromMaybe (Exception.fromString "Expected to find an H1 heading in this markdown.")
                    |> BackendTask.fromResult
            )


markdownBody : String -> BackendTask Throwable (List Block)
markdownBody rawBody =
    rawBody
        |> Markdown.Parser.parse
        |> Result.mapError (\_ -> Exception.fromString "Markdown parsing error")
        |> BackendTask.fromResult


findBySlug : String -> Glob String
-- ...

https://github.com/dillonkearns/elm-pages-v3-beta/blob/backend-tasks/examples/docs/app/Route/Docs/Section__.elm#L133-L164

elm-pages Exceptions compared to traditional exception handling

I'm not a fan of exceptions in other languages because of the way they introduce a new control flow mechanism, and because you can't tell which code could have a possible exception lurking, or whether the code above it is handling exceptions or not. There is also a strange back-and-forth where you could throw and catch exceptions at different levels and pass an error multiple directions.

With this abstraction, I think of this as "Managed Exceptions", just as many languages have side-effects while Elm has "Managed Effects." Similar to Managed Effects in Elm, Exceptions in elm-pages don't do anything in themselves, rather they need to be passed to the framework. Elm Managed Effects need to be passed to init or update, and elm-pages Exceptions need to be the final result of the error type variable in BackendTask Throwable value (where Throwable is just an alias to Exception ()).

The goal here is to give you explicitness of errors that come in vanilla Elm Tasks, but whereas regular Task's give you a Msg with a Result in it that you always need to explicitly handle, elm-pages BackendTasks give you a way to not handle certain error cases yourself but pass them to the framework to give an error status in the CLI or a server error code response for errors that you don't want to explicitly handle. So you have all the explicitness of error handling that we're familiar with in Elm and Task's, but the added convenience that we can let the framework give us nice error messages for cases that we don't expect and don't want to catch.

Performing all BackendTask's

Previously, DataSource's would cache all previous results in the chain of DataSource running, so if you hit an API it would never make that exact same API request twice but instead would just use the previous result.

This abstraction doesn't work for new effectful uses, such as elm-pages run scripts, or hitting effectful APIs from an action, where you want to be sure those effects are performed each time they're called.

It may take some iteration to figure out a nice way to allow for caching when the user wants it, for example to only make a given API request once and cache it any other time it's hit.

Feedback

I'd love to hear from you if you have thoughts on these APIs. Thanks for reading!

[wolfadex]

The idea of passing off the handling of an exception to the runtime sounds really nice! It immediately reminded me of my recent use of albertdahlin/elm-posix's Posix.IO.exitOnError which short circuits the IO steps.

---

As far as naming, I definitely like the move away from DataSource as it that seems to be only 1/2, more or less, of what it does now. Keeping it separate form the client side Task also makes a fair amount of sense for Elm I think given that a lot of Task related code is browser specific. If there were a way to make Task platform specific I could see going that route, but that doesn't seem possible or practical at least at the moment.

---

I do wonder, and may try to see, if I can use elm-pages as a cli for running my toy language instead of elm-posix. The idea of it possibly caching sounds intriguing for a cli tool, especially since I want to load data from both the file system and uris. Also sounds nice as I've been doing Advent of Code and I'm always nervous to do what some people do and hit the website directly to get the input instead of manually copy/pasting it.

[dillonkearns]

Good reference, thanks for linking to that.

Yeah, definitely similar in a lot of ways. The main differences are 1) the default output for certain framework-related errors, and 2) there's no way to erase the type variable while still having an exception.

There's some cost to that design because you have to pass around the extra wrapper type for Exception, but I really like being able to tell whether it's possible to emit an Exception or not by looking at the error in the type variable. For example, if you wanted to you could do something like ensureNoError : BackendTask Never value -> BackendTask Never value and write an elm-review rule that ensures that you use that everywhere you want to verify that you handle all possible errors (or just use the Never in the annotation certain places).

I'm still not 100% sure about the Throwable and Catchable types (the type alias), I might iterate on that a bit more. But other than that I'm feeling pretty happy with how this design makes things explicit, easy to tell where errors can happen, possible to handle errors, and yet fairly lightweight to just throw errors you don't care about without writing your own custom error formatter.

[synalysis]

Just a thought about the name BackendTask: wouldn't it be better named BuildTask? Backend is something I usually associate with a server running after build/deployment.

[wolfadex]

wouldn't it be better named BuildTask? Backend is something I usually associate with a server running after build/deployment.

These can run after build! It depends on the setup you're going for.

[dillonkearns]

Yeah, exactly right, with elm-pages v3 these can be resolved not just at build-time but also at request-time (for server-rendered Routes). Or it could be when running a script with elm-pages run as well. So build-time is just one context where these can run.

With elm-pages v3, the idea of there being both a frontend runtime and backend runtime for elm-pages will be a key concept that users will have to learn. BackendTasks are never resolved in the browser, always in whichever backend is powering it (CLI for build-time, serverless function if it's hosted that way, or it could even be run through a traditional server).

[synalysis]

Thank you for the clarification!