Addressing challenges with state machine reaction to animation state.

[Pyxus]

Main Issue

The issue, as I understand it, is the difficulty in having a state machine react to specific points of interest within an animation. Given that the state machine primarily exists to represent a combatant's state within a game, many state transitions are understandably linked to the animation state (such as exiting a state on animation end). However, Fray's state machine lacks a straightforward method for handling animation state reactions.

Problems To Be Addressed

Assuming I identified the main issue, these are the mains problems I think a good solution should address:

1. Selective Animation Reaction: It needs to be "opt-in" in some form. Such as encapsulating this logic into some kind of AnimationObserverState. Even if the main purpose is combatant state management, I still want this to be a general purpose state machine. Not every state will be concerned with the state of an animation.

2. Reaction To Arbitrary Points In Animation State: States within the state machine that are concerned with animations need to be able to access the state of an animation. I think knowing when an animation will end will be most common, but users may also want to be able to react to other arbitrary points.

3. Animation Tool Compatibility: Ideally the solution would support both the AnimationPlayer and AnimationTree. If I'm not mistaken, though the animation tree uses the animation player I think the playback is handled within the tree. Meaning if only the player is observed then animation state wouldn't be properly reported. I may also want to consider supporting AnimatedSprite.

Conclusion

As of now, I'm just trying to full grasp the problems users have with the state machine. I don't have a definite solution in mind yet. Please let me know if I correctly understood the limitations, and whether you believe solving the mention problems would address those limitations.

Discussion Inspired By These Comments

From @Remi123

I think the only thing missing is some helpers for transition using a signal. Animations have a lot of points of interest like on_hit and I'm not sure what is the canonical way of interacting with it using FrayStateMachine. For now it's either something that I manage within the FrayState, or whenever the on_hit signal is trigger I send a BufferedInput through the FrayBufferedInputAdvancer , which is bad because it's buffered in some cases. It would be nice to connect Timer's Signal with this.

From @Remi123

I've implemented a relatively simple state designed to play an animation and await either the end of the anim, or a signal. In fact all "actions" in the video is using this state. I know you will create a discussion about this but here is my pseudo-code on this state so you can think about it:

# The signal can_advance is set in the animation
var ended :bool = false
func _is_done_processing_impl() -> bool:
  return ended
func anim_finish(_anim_name = ""):
  ended = true
  get_root().advance()

func _enter_impl(args:Dictionary):
  ended = false
  if not statemachine.can_advance.is_connected(on_anim_finish):
    statemachine.can_advance.connect(anim_finish,CONNECT_ONE_SHOT)
  if not animationplayer.animation_finished.is_connected(on_anim_finish):
    animationplayer.animation_finished.connect(anim_finish,CONNECT_ONE_SHOT)
  animationplayer.play(args.anim_name)

func _exit_impl() -> void:
  ended = false
  # Make sure we disconnect signals.
  if statemachine.can_advance.is_connected(anim_finish):
    statemachine.can_advance.disconnect(anim_finish)
  if animationplayer.animation_finished.is_connected(anim_finish):
    animationplayer.animation_finished.disconnect(anim_finish)

From @Kuenaimaku

I'm in agreement with Remi123 on helpers for transitions. Outside of rolling my own FrayState I don't see a great way of making it so the animation can control when a state can be considered "finished processing", which makes sense given how the intent is for these 3 modules to act independently of each other.

What I'm aiming to do kinda already goes against keeping these 3 modules separate, as I'd like to annotate my animations with all these properties (what FrayState is linked to a HitState? At what point can this animation be considered finished processing to return back to the idle state? at what point can a player input a button to 'cancel' into another action?) rather than having it as part of the state setup.

[Pyxus]

After investigating it a bit AnimationTree does indeed handle the playback. Also, since the tree can blend together different animations which play at different rates I think the best way to handle reacting to the animation tree is to somehow have let the user react to specific animations that compose the tree. That way they can have a sort of "root" animation which everything is synched around regardless of how much they blend animations.

That being said I still need to figure out if it's possible to fetch the state of animations within the tree.

[Remi123]

I've been thinking about this problem for some time and the resource are somewhat scarce but the few that exists are descriptive.

For example, as a good overview of an animation graph, I can only suggest this video by Bobby Anguelov : https://youtu.be/R-T3Mk5oDHI and there is also a video on Animation on his channel in Live Playlist,but I doubt you need it. He is a Principal Programmer at Valve and has a decade of dealing with Animation, network and all sort of thing. I think he give very good overview of what your library should do, which is basically to treat any AnimationPlayer or AnimationTree as a blackbox.
He is also opinionated which I think make any informative video he made very funny.

The AnimPlayer/Tree should be treated as a blackbox because it create a create a separation of concern. If the state machine is in a state but playing animation is wrong, it's the AnimPlayer/Tree problem, not the state machine. The AnimPlayer/Tree can help by giving back some information on timing and distance ( think root motion ), but it's mostly an interface problem.

To make it short, I think any state machine should provide some sort of interface to Signals and a way to propagate them to the current state, and only the current state. You'll find statement very similar to how the collision library is implemented, well that's because it's a good pattern. I'm advocating for signal management inside the state machine.

The issues is that I can create signal with arbitrary name and with arbitrary arguments and I don't know what interface to add to the FrayState. In collision it's easy since on_intersected(FrayHitbox3D ,FrayHitbox3D ) is always dealing with those two types, but now it's a lot more. And how the builder will interface with that is unclear yet.

I'm mainly talking about Signal inside State, but I think adding a transition_signal will be useful to people, which take a couple of signal and the first signal that trigger advance the state. And maybe add an option to specify the number of signals to trigger before advancing, with -1 being all of them, and 1 the default. 0 is just advancing the state. You can see in my example that it's basically what I'm doing.

Signal introduce timing management, so it's possible to use await and Tween inside a state. This should be embraced, especially inside enter_impl. My reach state in my code for ladder is basically awaiting a Tween to finish going to a specific place, then await for a specific animation to finish while moving with root motion. How to help with using await and Tween, I have no idea. Maybe add a boolean inside FrayCompoundState default to true named ended and is_finish_processing_impl is always returning it, so my Tween can set it to false in enter_impl, tween a bunch of stuff, then tween_callback(func():ended = true) at the end. I'm brainstorming.

[Remi123]

I noticed the tree still calls functions keyed in the AnimationPlayer, so I could modify the animation tracks once at runtime to insert calls to a private function every n steps. Then states could be made to react to the nth step of specific animations. In fact, that would support both the tree and player. Though this is just me brainstorming, there are likely some issues I've yet to consider.

I may not have well understood what you want to do to interface between FrayStateMachine and AnimationPlayer. Personally my pain points are dealing with state that wait for specific part of the animation to occurs, so signal are very core to my solution. I think I need to understand better what you think of going.

After rereading your statement about AnimationObserverState, it would be some kind of State that wrap the player/tree, contains the logic to start the animation, and react when reaching n step, probably defined by the builder? Except the n step, it's very similar to my setup, but I'm awaiting for a signal. I want to add that modifying an animation at runtime can be finicky, as the AnimationPlayer/Tree cache it's data before using it so you might fight the backend, at least I didn't succeeding in creating new animation at runtime but I was doing something mure more heavy.

Or maybe the n steps are things like "Startup frame", "Active Frames" and "Recovery frame" instead of frame number ? I need more detail :P

[Pyxus]

I may not have well understood what you want to do to interface between FrayStateMachine and AnimationPlayer. Personally my pain points are dealing with state that wait for specific part of the animation to occurs, so signal are very core to my solution. I think I need to understand better what you think of going.

It's late so i'll re-read and properly read your whole post when I'm more awake. But I wanted to try and clarify a little that my goal is to share the animation state with the current fray state. Specifically "What animation is playing?", "How long is the animation?" and "At what frame is the animation currently on?" regardless of if that is represented in literal frames or seconds.

My thought with modifying the animation player is that I would edit each Animation resource to add a method track which keys a call to a private method, lets say for explanation sake... "_animation_update(name, length, frame)" every n steps/frames. And that method would be passed the name, length, and frame of the current animation as arguments. This track modification would only need to be done once per animation during the state machine setup.

This would only really need to be done to support the AnimationTree node as both AnimationPlayer and AnimatedSprite appear to have easier ways of determining the animation state information I care about. I think this also work around any manner of tree blending since no matter how you speed up, loop, or combine animations each will independently call the method at whatever rate they're playing and the pass long the necessary information.

Again though I'm just brainstorming and making sure I cover the limitations you and the other commenter encountered 🤔

[Pyxus]

@Remi123 I quickly put together what i'm currently calling "AnimationObservers" on a new branch. I'm still brainstorming but maybe taking a look at it will help you better understand the approach I'm leaning toward? https://github.com/Pyxus/fray/tree/state-anim-reaction

If I go through with this then the only other issue to address would be "How do states subscribe to observer events". But i'm tempted to leave that to the users... You could pass the observer through the new context dictionary and have your custom states subscribe as needed. Though i'm still trying to understand how much this helps the original problems brought up.

Edit: Actually after re-reading, I likely am misunderstanding 😅. The observer is an interesting idea on it's own but im not so sure this is address everything. I'll have to give this more thought.

[Remi123]

Ok so I've read and reread the changes in the branch, and it's starting to make senses.

So here is my brief resume, just to make things clear:

# A state machine node can have subnode of types FrayStateMachineComponent.
FrayStateMachine
|-FrayAnimationObserver
|-FrayBufferedInputAdvancer

The FrayAnimationObserver can have a resource of type AnimationTracker, and this is where you interface with the Animator depending on his type. AnimationPlayer does something, Tween also get their tracker, etc.

I've read your help wanted, and I think it's possible to reference a resource's method using double colon like this var node_path = NodePath("Path2D/PathFollow2D/Sprite2D:texture:load_path"). I've found this in the doc of NodePath in the function get_subname. Maybe you can try something with this but it's outside my expertise.

My current use case is I'm holding all my signals in a built-in script on my StateMachine node and calling emit_signal("X") in the method track with path NodePath("%FrayStateMachine"). It's a little bit tedious to subscribe to a signal inside a State since it's a resource but I've seen you added connect_while_active in state which is really appreciated. One things to add is a simple get_state_machine() from a FrayCompoundState and it would be perfect.

In terms of functionality, every AnimatorTracker seems fine, except the one for AnimationTree. Adding events every frame in every animation is just too much and doesn't really help knowing which state you are. While blending between multiple animations, every animation that is running will trigger the event and this can cause confusion if you are using the same animation multiple time. Especially problematic with additive animations. So what is really useful is which node is currently running.

Unfortunately, AnimationTree have the tree_root property that is an AnimationNodeRoot, and it become a graph. I'm saying unfortunately not because you have to explore each nodes, check the type and explore subgraph if possible, but because some nodes have better interface than other. BlendTree have the node_changed signal but I don't know if it's used at runtime or to construct the tree. AnimationNodeStateMachine has a AnimationNodeStateMachinePlayback that you obtain from AnimationTree,but it doesn't contain any signals so you have to check every frame if current_node has changed. It become a mess very quickly.

AnimationTree itself has animation_started and animation_ended signal, but I remember the author being very reluctant to add this since it goes against what this node is about. I'm not saying AnimationTree is bad. It's a really good animation blender, but the interface could be rethought and I think there is a proposal for that somewhere.

I was very busy this weekend so didn't test the new state machine, but I like what see thus far. Continue the good work

[Pyxus]

Addressing Animation Tool Compatability

Here are my current thoughts on addressing this problem.

Supporting AnimationPlayer

Very straight forward. The player can tell us the current animation, it's length, and it's position in the playback. That's everything we'd need to know to support it.

Supporting AnimatedSprite

Similar situation to player. The sprite can tell us the current animation, current frame, frame count, and the FPS of the animation.

Supporting AnimationTree

The only solution I've thought of to support this is to insert function calls into the animation player at runtime. More details here:
#73 (reply in thread)