Concerning support in RL for Relative Position Control & Compatibility with BSK MultiSatBskSim framework

[Thomas-Chan-2019]

Hey there!

I wonder if there would be a possibility for the BSK-RL side to support RL/training for close-proximity relative position control between chief-deputy spacecraft pairs, whereas using a similar scenario creation format as how it is done with the MultiSatBskSim framework such as from this scenario.

I have been using MultiSatBskSim to implement relative position control in which I find it very scalable, and now I am looking to extend some work with BSK-RL! As I read from #217 that agent actions to control thrust is under development, I am trying to branch off from feature/144-relative-motion and looking to create a child FSWModel to begin with from bsk_rl/src/bsk_rl/sim/fsw.py. Would this overlap with some of your ongoing development in this branch or would there be anything I need to be cautious about?

I also notice that the structure here is fairly familiar compared to the MultiSatBskSim structure. I understand that there are specific Gymnasium-required formats and it might be hard to achieve full compatibility, but it would be great if users can create mission scenarios in a similar approach as compared in MultiSatBskSim.

Just wish to raise a possible discussion here, would really appreciate if there are suggestions, thank you!

[Mark2000]

Yeah, you've found the two main issues that touch on that, #144 and #217. I would recommend not directly forking off the feature/144-relative-motion branch since pieces of it are being cleaned/renamed and brought in part-by-part, as noted int #144. In particular, #250 will be merged by the end of the week and provides the high-level framework for continuous action spaces, and #245 and #246 have added some other useful features for relative motion.

Depending on the exact type of control you want to do, you'll probably need to implement a new FSW and continuous action model, which the feature/144-relative-motion branch does have examples for with impulsive burns. The trickiest part is the data/rewarder system, but that branch could also provide a starting place.

Ultimately, yes, this is pretty similar to the MultiSatBskSim structure but there are some differences. Could you specify what features from that you'd like to see within BSK-RL?

[Thomas-Chan-2019]

Hey Mark,

Thank you for the guidelines! I guess I should dive through the reward creation parts first and figure out what I can do, I might want to do thruster control for close proximity relative position control between cheif-deputy pairs. Since you mentioned that I shouldn't fork off from feature/144-relative-motion, and now I see that #250 has been merged, do you suggest that I branch off via develop here?

Regarding BSK-RL features, right now I wasn't thinking of any specific features here since I feel like it might be too specific to serve a broader use case. I have been thinking if using a user-defined scenario configuration file (such as a JSON/YAML) would scale up scenarios easily, for example an attempt on my fork to take in initial conditions & Hill-frame static setpoints along chief-deputy spacecraft pairs:

{                                                               
    "targetOE": [6578000.0, 0, 45.0, 48.2, 347.8, 85.3],
    "InitConfig": [
            {
                "isTarget": 1,
                "scName": "Astrobee",
                "sigma_BNInit": [[0.1], [0.6], [-0.8]],
                "omega_BN_BInit": [[0.0], [0.0], [0.0]],
                "init_dr_hill": [0.0, 0.0, 0.0],
                "init_dv_hill": [0.0, 0.0, 0.0],
                "target_dr_hill": [0.0, 0.0, 0.0],
                "target_dv_hill": [0.0, 0.0, 0.0],
                "oe":[6578000.0, 0, 45.0, 48.2, 347.8, 85.3]
            },
            {
                "isTarget": 0,
                "scName": "Astrobee",
                "sigma_BNInit": [[0.0], [0.0], [0.0]],
                "omega_BN_BInit": [[0.0], [0.0], [0.0]],
                "init_dr_hill": [-1.0, 5.0, 0.0],
                "init_dv_hill": [0.0, 0.0, 0.0],
                "target_dr_hill": [3.0, 2.0, 10.0],
                "target_dv_hill": [0.0, 0.0, 0.0],
                "oe":[6578000.0, 0, 45.0, 48.2, 347.8, 85.3]
            },
            {
                "isTarget": 0,
                "scName": "Astrobee",
                "sigma_BNInit": [[0.0], [0.0], [0.0]],
                "omega_BN_BInit": [[0.0], [0.0], [0.0]],
                "init_dr_hill": [0.0, -0.5, 0.0],
                "init_dv_hill": [0.0, 0.0, 0.0],
                "target_dr_hill": [0.0, 2.0, 0.0],
                "target_dv_hill": [0.0, 0.0, 0.0],
                "oe":[6578000.0, 0, 45.0, 48.2, 347.8, 85.3]
            }
    ]
}

This is however a way-too-specific use case, and I don't see how it could be implemented on the BSK-RL side in a useful manner given that lots of other parameters can be considered in training or hyperparameter tuning... I might instead raise a discussion on the BSK repo to check whether this kind of scenario creation approach could be beneficial.

Lots of thanks again!

[Mark2000]

Yeah, the current develop should be a good starting point. Depending on what your reward looks like it could be easy or hard to implement, but if you have an idea of what the objective is I'd be happy to provide some thoughts on how to implement it.

Also, you might be interested in this paper we presented recently, covers a similar use case: https://hanspeterschaub.info/Papers/Stephenson2025a.pdf