Solved – If I use dropout in a neural network and run it for a large number of steps, do I risk deleting all the units

I think you're conflating the dead ReLU problem and dropout. ReLU nets often use dropout, but they are not the same.

Dropout

Using dropout "freezes" some units (at random and usually but not always independently) by ignoring their weights at each iteration. The frozen units are not set to zero, but for that iteration, the network pretends that they are zero. The frozen units are not updated for this iteration.

For the next iteration, all units, including the ones that were frozen during the last iteration, are available to be frozen (again, at random). It's possible for some units to be frozen several times in a row (albeit with diminishing probability).

The non-frozen units are evaluated and updated as usual.

You can use dropout with any type of neuron.

Dead ReLU

So-called "dead neurons" or "dead ReLUs" happen when the weights update in such a way that the unit always returns 0 for any of the inputs in the data set; for example, if the inputs are between 0 and 1, but the weight is negative, then the ReLU always returns 0. Once this happens, the back-propagation can't update the unit: because it's output is always 0, so is its gradient, and the weight never changes. ELUs and other variations on ReLU can ameliorate this.