-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
action distribution for estimating V #8
Comments
Thanks for your question. We use uniform sampling because there is no direct way to evaluate the log-probabilities of action of SVGD policies, which would be needed for the importance weights. Using some other tractable policy representation could fix this issue. You're right that uniform samples do not necessarily scale well to higher dimensions. I haven't really studied how accurate the uniform value estimator is, but from my experience, using more samples to estimate the value improves the performance only marginally. |
ok, i see. |
I could be totally misunderstanding, but doesn't appendix C.2 talk about how one can use the sampling network for q_a' and derive the corresponding densities so long as the jacobian of a'/epsilon' is non-singular? |
I see, that's indeed confusing. You are right in that we could compute the log probs if the sampling network is invertible. My feeling is that, in our case, the network does not remain invertible, and that the log probs we would obtain that way are wrong. We initially experimented with this trick (and that's why we discuss it in the appendix), but in the end, uniform samples worked better. We'll fix this in the next version of the paper, thanks for pointing it out! |
My pleasure! Glad I was sort of on the right track. That's very interesting, especially since non-singular weight matrices or choice of activation function are the only thing off the top of my head that might make a feedforward net non-invertible. I might play around with that. |
Also in "softqlearning/softqlearning/algorithms/sql.py" |
Do you mean the expectation over states and actions in Eq. (11)? It is OK, since the corresponding gradient estimator is unbiased, though it can have high variance. |
I see. |
Hi,
First of all, thanks for this inspiring work!
In
softqlearning/softqlearning/algorithms/sql.py
Line 164 in 59c0bbb
it seems to me that action is sampled from uniform distribution when estimating
V_{soft}
.In Sec. 3.2. of your original paper, it is stated that
have you experimented with sampling from current policy to estimate
V
? Or, how good does uniform distribution do in practice, especially in higher dimensional cases?thanks,
The text was updated successfully, but these errors were encountered: