Explicit distribution of transition and reward

[pkel]

I'm trying to implement the Selfish Mining problem defined here https://arxiv.org/abs/1507.06183.

The problem has explicit (SparseCat) state transitions and defines the reward for each transition. I have a function of the following form.

transition_and_reward = function (s, a)
  return SparseCat( [ (sp_0, rew_0), ..., (sp_n, rew_n) ], [prob_0, ..., prob_n] )
end

How would you implement this in POMDP?

After reading the doc, it seems I have to choose from the following:

1. Implement transition with SparseCat return value and reward returning an expected reward. This implies loosing some information about the association between transition and reward.
2. Implement transition with SparseCat return value and include the reward in the next state. Then make the reward = function (s, a, sp) function read the immediate reward from sp. This approach implies that reward = function (s, a) remains undefined. It also causes redundant states.
3. Implement the gen function. This implies loosing the explicit distributions.
4. Implement different POMDPs for different solvers.

Each has its own shortcomings. Do you have a better solution?

Discussion #509 seems to be related, but I do not understand the proposed solution. What is "a compositional approach"? Does it solve my problem?

Bonus questions:

1. What speaks against defining (PO)MDPs with the combined transition_and_reward function presented above?
2. What about info values? I actually have a SparseCat for (sp, rew, info). The info is important to transform the MDP later down the pipeline, so I'm keen not to loose it.

[zsunberg]

Hi @pkel ,

I think the best way to handle this is to implement both the gen function and transition with (s, a) arguments. You can implement both simultaneously; you just have to make sure that reward returns an expectation consistent with gen (which is why implementing both separately is usually not recommended). Then gen will be used in simulations and transition and reward will be used in solvers that need explicit representations.

To ensure reliability, you could pre-compute transition probabilities and reward expectations for all states using your current transition_and_reward function.

[pkel]

Excellent. I will try this approach, thanks!