Tricks of Sigmoid Function

During my research of Bayesian Deep Models (integration of Bayesian graphical models with deep learning models), I found several handy tricks when dealing with sigmoid functions. Here, I summarize several for future use and also for other researchers who might find it useful.

Variational Lower Bound on Sigmoid $\sigma(x)$

Expectation of Sigmoid function with Normal distribution

Consider the following logistic-normal integral:

The logistic-normal integral does not have analytic expression. However, for mathmatical simplicity, we can approximate the expectation. In the end, we will demonstrate that the integral is approximately a reparameterized logistic function.

First, we can approximate the sigmoid function with a probit function.

A little fact is that the probit-normal integral is just another probit function:


It actually means, given a normally distributed random variable $x$, the sigmoid of $x$ is approximately the sigmoid of $\mathbb{E}[x]$ with some adjustment by the variance.

Some others