Sorry for taking so long to reply. 1) On paper, yes certainly there is some mathematical rigour. But I don’t see Bayesian models winning kaggle. Basically it would struggle if you required it to spit out point estimates. 2) I get that they integrate out all possible values of regularizers but I have seen so many instances where choice of prior did matter.

Just lastly though I should mention that I do think that Variational Bayes methods might actually make an impact in the data science field in the near future. So I still ❤ PyMC3 which has strong VB support.