Implemented Solvers for Finite Horizon POMDP Problem

[turguy]

Hello everyone,

I am conducting a research project and have developed a 7-tuple for my POMDP problem, and implemented the problem using the Julia POMDP interface (QuickPOMDP). Using solvers such as SARSOP or Point Based Value Iteration, I was able to obtain a policy for the infinite horizon POMDP.

However, I now need to convert this problem into a finite horizon POMDP and obtain a policy for this as well. To achieve this, I am using the "FiniteHorizonPOMDPs.jl" package and testing with a horizon length of 2. With just one line of code, I can change the infinite horizon problem to a finite horizon problem (finite_POMDP = fixhorizon(infinite_POMDP, 2)).
But I am encountering difficulties with obtaining a policy using this method. When using SARSOP, I received an error message: ERROR
model.pomdpx:Line 129076:
In Observation Function observation : there are missing transitions
Transition not specified for instance a0 s0
.

And when using the Point Based Value Iteration solver, the model did not load even after running it for a day. This is surprising, as with the same setup for the infinite horizon, I was able to obtain policies using these solvers in less than an hour (for testing purposes, I minimized the size of the problem).
I would greatly appreciate any suggestions or ideas on what I may be doing wrong.

[qhho]

I believe both the implementations of the Julia wrapper for SARSOP in C++ and Point Based Value Iteration do not currently work well with finite horizon POMDPs. For finite horizon MDPs, there is https://github.com/JuliaPOMDP/FiniteHorizonValueIteration.jl

Edit: Could you try using NativeSARSOP (https://github.com/JuliaPOMDP/NativeSARSOP.jl) and see if the issue persists?

@WhiffleFish perhaps we can add functionality for the finite horizon case for NativeSARSOP

[qhho]

Sorry, I was mistaken, Point Based Value Iteration and NativeSARSOP should work in general with finite horizon POMDPs. There is an issue with SARSOP.jl regarding the model.pomdpx file that seems to happen for all finite horizon POMDPs. This could be a bug.

[qhho]

@turguy Could you share more about the specific POMDP problem you have implemented?

[turguy]

Hi, thank you for all the replies. Here is the notebook that I used for my POMDP problem

[WhiffleFish]

Hey @turguy, FiniteHorizonPOMDPs.jl has been fixed, so you should be able to solve your finite horizon POMDP now.

[turguy]

Thanks for letting me know! I appreciate the update. You're the best!