Solved – What exactly is a hyperparameter

A hyperparameter is a parameter for the (prior) distribution of some parameter.

So for a simple example, let's say we state that the variance parameter $tau^2$ in some problem has a uniform prior on $(0,theta)$.

(I personally would be unlikely to do such a thing, but it happens; I might in some very particular circumstance)

Then $tau^2$ is a parameter (in the distribution of the data) and $theta$ is a hyperparameter.

If we then in turn specify a (prior) distribution for $theta$ (e.g. that it's Gamma with mean 100 and shape parameter 2), that's a hyperprior – a prior distribution on a parameter of a prior distribution.