Error in solving QuickMDP using DiscreteValueIteration

[Manavvora]

Hi,

I am trying to solve an MDP declared using QuickMDP where I've explicitly declared n_states = 101*102 = 10302. However, when I try to solve the MDP using DiscreteValueIteration, I get the following error message:

Warning: Problem creating an ordered vector of states in ordered_states(...). There is likely a mistake in stateindex(...) or n_states(...).
│
│ n_states(...) was 20604.
states corresponding to the following indices were missing from states(...) : (A list of indices here)

I have not explicitly defined the stateindex in the QuickMDP.

This is how I have defined the MDP:

a = [],
m = QuickMDP(
actions = [0, 1],
#obstype = Tuple{Int64, Int64},
discount = 0.95,
a = [],
temp = (),
for i in 0:101
for j in 0:100
push!(a, (i,j))
end
end,
states = a,
statetype = Tuple{Int, Int},

n_states = 102*101,

transition = function(s,a)
    if a == 1
        return Deterministic((100, s[2] + 10))
    elseif a == 0
        if s[1] - 2 < 0
            return Deterministic((101, s[2]))
        elseif s[1] - 5 < 0
            return SparseCat([(s[1]-2, s[2]), (101, s[2]), (101, s[2])], [0.05, 0.80, 0.15])
        elseif s[1] - 10 < 0
            return SparseCat([(s[1]-2, s[2]), (s[1]-5, s[2]), (101, s[2])], [0.05, 0.80, 0.15])
        else
            return SparseCat([(s[1]-2, s[2]), (s[1]-5, s[2]), (s[1]-10, s[2])], [0.05, 0.80, 0.15])
        end
    end
end,

reward = function(s,a,sp)
    if sp[2] > 100
        return -100
    end
    if sp[1] == 101
        return -50
    elseif sp[1] > 0
        return 3
    end
end,

initialstate = Deterministic((100, 0))

)

Edit 1: The previous error has been sorted but the new error I am facing is this:
KeyError: key (100, 101) not found

[WhiffleFish]

Hi,

This appears to be because your transition function is allowing transitions outside of your defined state space.

s = (100,91)
s ∈ states(m) # true
a = 1
tdist = transition(m, s, a) # deterministic
sp = tdist.val # (100,101)
sp ∈ states(m) # false

[Manavvora]

A quick follow-up :
The new error that I'm getting is this: MethodError: no method matching +(::Nothing, ::Float64)

Also, the output I'm getting after defining my QuickMDP is:
QuickMDP{nothing, Tuple{Int64, Int64}, String, NamedTuple{(:stateindex, :isterminal, :states, :statetype, :discount, :n_states, :actions, :a, :actionindex, :transition, :reward, :initialstate), Tuple{Dict{Tuple{Int64, Int64}, Int64}, Bool, Vector{Any}, DataType, Float64, Int64, Vector{String}, Vector{Any}, Dict{String, Int64}, var"#5#7", var"#6#8", Deterministic{Tuple{Int64, Int64}}}}}((stateindex = Dict((41, 10) => 4193, (17, 12) => 1747, (19, 86) => 2025, (65, 69) => 6700, (97, 68) => 9963, (55, 35) => 5646, (9, 80) => 999, (16, 99) => 1732, (31, 14) => 3177, (19, 16) => 1955…), isterminal = false, states = Any[(0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8), (0, 9) … (101, 92), (101, 93), (101, 94), (101, 95), (101, 96), (101, 97), (101, 98), (101, 99), (101, 100), (101, 101)], statetype = Tuple{Int64, Int64}, discount = 0.95, n_states = 10404, actions = ["do-nothing", "maintain"], a = Any[], actionindex = Dict("maintain" => 2, "do-nothing" => 1), transition = var"#5#7"(), reward = var"#6#8"(), initialstate = Deterministic{Tuple{Int64, Int64}}((100, 0))))

As can be clearly seen, the MDP is not allotted a UUID. Is this causing the error? If so, how can I fix this?

Edit: The MDP is now being allotted a UUID but the error still persists. And I am also facing a similar error for the QuickPOMDP version of this where it says MethodError: no method matching *(::Float64, ::Nothing)

[zsunberg]

The + error is most likely caused by your reward function returning nothing. This can happen if you have some if statements that do not have associated elses and there is not return 0 at the end of the function. In the code posted above, your reward function will return nothing for example if sp[2] < 100 and sp[1] < 0.

[Manavvora]

Okay thank you! So I'm guessing the * error would also have been caused by a similar thing for the QuickPOMDP right?

[zsunberg]

Yes, usually when I see an error with a mathematical function trying to do something with a Float64 and a Nothing I suspect that kind of error.

[Manavvora]

Okay got it! Thanks a lot!