For the question
I previously posted here with a general question about elo ranking applied to a a bit weird sport. From the ranking POV it's a bit like marathon – you can have up to n players but – unlike i.e. chess – your final ranking is based on single performance on your part and final result is based on the number assigned to your single performance.
Is it important? Kind of.
I could represent this sport as n^2 – n pairs. This sucks a bit because it makes a lot more important for your elo ranking to win massively populated tournaments than to win several small. It may inflate one's ranking based on just one-time fantastic result that just happened to coincide with a massively populated run.
On the other hand one can take the average of all the results and make each tournament worth just like one tournament (one run after all). This however leads to small fluctuations of the ranking if the competitions are not really that common (Which is a case in my case). You can offset it by big K parameter but that might lead to very volatile ranking. Also it feels that 2 competitor tournament shouldn't be worth just as much 20 people tournament.
After trying both of the above approaches I settled for the latter but with variable K – each competition has K growing with the amount of competitors. So 1/3 of K is static and 2/3 depends on the number of competitors. This kind of looks OK (OKish maybe) but I still have my doubts.
And to the question: how reasonably verify that you've chosen correct K value? So far I just build a ranking, plotted it and looked for something I found weird. But that doesn't sound mathematical. Or maybe variable K is not good solution?
I'm aware I can apply suggestion from here: How to prove that Elo rating or Page ranking have a meaning for my set? but that's not completely what I'm asking about.
As nobody suggested a better solution I went with this. I discussed it briefly with professional mathematician and he didn't discard the solution.
I assumed that – in above explained approach – the K should be something like this (note – x is a number of competitors in a given competition so K it's not constant) $$ Ax^2 + Bx + C $$ Then I worked on my data set through various values and assumed that best chosen K should have minimal differences between expected result and actual result (makes sense, doesn't it?) data-set wide. It turned out that best polynomial on my data set had $A=0$ and $C=0$. This means that I actually could treat the whole tournament as round robin after all.
In my case the optimal value was $B=42$ (I required for the value K to be represented by natural numbers) which is nice but only coincident.