Solved – Tensors in neural network literature: what’s the simplest definition out there

For the purposes of data analysis, you can effectively consider them as arrays, possibly multidimensional. Thus they include scalars, vectors, matrices, and all higher order arrays.

The precise mathematical definition is more complicated. Basically the idea is that tensors transform multilinear functions to linear functions. See (1) or (2). (Multilinear functions are functions which are linear in each of their components, an example being the determinant considered as a function of column vectors.)

One consequence of this mathematical property defining tensors is that tensors transform nicely with respect to Jacobians, which encode transformations from one system of coordinates to another. This is why one often sees the definition of tensor as "an object which transforms in a certain way under changes of coordinates" in physics. See this video for example, or this one.

If we are dealing with sufficiently "nice" objects (all of the derivatives we would like to exist and well-defined are), then all of these ways of thinking about tensors are essentially equivalent. Note that the first way to think of tensors which I mentioned (multidimensional arrays) ignores the distinction between covariant and contravariant tensors. (The distinction is with regards to how their coefficients change under a change of basis of the underlying vector space, i.e. between row and column vectors essentially.) See these other StackExchange questions: (1) (2) (3) (4)

For a book used by researchers studying applications of tensors to neural networks (for example at Technion in Israel), there is Wolfgang Hackbusch's Tensor Spaces and Numerical Calculus. I have not read it yet myself, although some of the later chapters seem to use advanced mathematics.