Different policy types with particle filter

[DonalOCois]

Hi all,

I am new to Julia and POMDPs, so I'm sorry if this is a very simple matter with which I'm now wasting your precious time!
I was playing around with the code of the video tutorial for the particle filter. In that code, the particle filter is given a random policy, and I wanted to see if I could apply a function policy instead.

I tried to write a very simple discrete action space which is just:
actions = [1,-1],

And then have one of the coordinates of the next generative state space include one or the other of those two actions:

y′ = clamp(s[2] + a + randn(), 0, 30)

And then finally the policy just says that if the second element of the state space rises above 10, use action 2, otherwise, use action 1:

begin
policy = FunctionPolicy(s->s[2] > 10 ? a->a[2] : a->a[1])
end

That was pretty much the extent of the idea. I replaced the random function with this function policy, but unfortunately my particle filter didn't take kindly to this as seen below. Any advice here would be very much appreciated!

history =
MethodError: no method matching getindex(::ParticleFilters.ParticleCollection{Tuple{Float64, Float64}}, ::Int64)

[zsunberg]

Hi @DonalOCois , welcome to the Julia POMDP ecosystem.

It looks like your problem is in the FunctionPolicy. First, just to make sure you understand the syntax, you may want to review the docs about anonymous functions: https://docs.julialang.org/en/v1/manual/functions/#man-anonymous-functions . In particular, right now, the function in your FunctionPolicy returns a function (either a->a[2] or a->a[1]) when it should just return an action (either -1 or 1).

Additionally, if you are using a particle filter, you should be defining a policy that acts on beliefs, not states. This means you should use the distribution interface or particle belief interface to interact with the belief. Since you are dealing with a belief, s[2] > 10 is ill-defined since there can only be some probability that s[2] > 10. Suppose that you want to take action -1 if the mean of s[2] in the belief is greater than 10, you could implement it like this:

FunctionPolicy(b -> mean(b)[2] > 10 ? -1 : 1)

Hope that helps! I'm not very familiar with the videos, so perhaps if you point to a video and give the timestamp, we can offer better-contextualized help.

[DonalOCois]

@zsunberg Yes - that worked nicely! Thank you!
So I think I understand the function well enough in principle, although I have a lot to learn about the syntax.
A few questions arising from what you've shown me, if that's ok:
First, I noted that as you mentioned I can ask the function to check if "any" (or indeed "all") of the particles are above the threshold here, but not whether the mean of the particles are above it. The error given in that case is that a Boolean should be returned, and not a Float64 - but isn't the answer to this logical question (using "<") a binary value?
The second thing was that it seems a little strange to me to just leave the discrete action space as [1, -1] for example, and calling them as such, rather than labelling them as 'do something' and 'do something else' - and then calling them as a[1], a[2] or by these assigned names. Is this considered ok, or could my presentation and calling of those actions be improved here?

Oh and finally one more thing that occurred to me: If I wanted to return the results of additional questions, how might that look in terms of syntax?

For example, here we had two possibilities being returned, 1 or -1:
FunctionPolicy(b -> any(s[2] > 10 && s[2] < 20 for s in particles(b)) ? -1 : 1)

But what if we wanted to return a third action, resulting from a different state space? Presumably it would involve asking a second question (if the state is between 20 and 30, then take an a different action than the two we listed already).

Thanks again very much for taking the time

[rejuvyesh]

@DonalOCois FunctionPolicy can take any function as argument. So a cleaner version of above would be:

function mypolicy(b)
  if any(s[2] > 10 && s[2] < 20 for s in particles(b))
    return -1
  else
   return 1
  end
end

FunctionPolicy(mypolicy)

which hopefully makes it more obvious in terms of how to extend this.

[DonalOCois]

@rejuvyesh yes, that makes it very clear - thank you! That's much more familiar to me.
I am still curious as to why "any" is acceptable but not "mean" for example. It would just be interesting to know.
Also the actions that can be taken are independent of the action space, which I would have thought defeats the purpose of having an action space?

[zsunberg]

I am still curious as to why "any" is acceptable but not "mean" for example. It would just be interesting to know.

mean(b) is roughly equivalent to sum(particles(b))/length(particles(b)) and sum(particles(b)) requires a way to add elements of particles(b) together. But, there is no defined way to add Tuples together. You could, on the other hand, do mean(p[2] for p in particles(b)) because that is just adding numbers together. any doesn't require adding anything together.

Also the actions that can be taken are independent of the action space, which I would have thought defeats the purpose of having an action space?

Right now, you are defining a heuristic policy with your own human knowledge of the action space, so you are hard-coding actions like -1 and 1. Your intuition is correct that this is not a good general approach if you want to handle a wide range of POMDPs - The policy you define in this way will only work for POMDPs that understand actions -1 and 1. Writing a solver designed to handle a wide range of POMDPs is a different type of task and again, your intuition is correct that you would want to use the action space of whatever POMDP is given to your solver. To see an example of this, check out this part of the documentation: https://juliapomdp.github.io/POMDPs.jl/stable/offline_solver/#Example:-Defining-an-offline-solver

[DonalOCois]

@zsunberg @rejuvyesh Thank you both for those very detailed answers.
The whole issue of using a 'function policy' is now a lot more clear for me, so I'll mark this thread as answered. Appreciate your patience and willingness to help very much.

The concept of the solver to handle more general POMDP cases is definitely the next logical step for me to look into, so I'll read the documentation and try to experiment with that!