Yahoo! Labs

Q. Why does the challenge include separate validation and test sets?

A. When you submit a prediction, you get immediate feedback on the validation set, so you have an idea of how well your algorithm is doing. Also, you can compare yourself to other competitors on the leaderboard. The validation set can in fact be thought of as a test set for "fun". No feedback on the "real" test is provided. This is to prevent competitors from optimizing against that set (that is, bouncing the output of an algorithm against the test set over and over to see which output gets the highest score). So the validation set is sort of a practice set; the test set is the only one that counts for purposes of your score in the competition.

Q. Why does the challenge restrict the number of submissions per day?

A. For two reasons. First, you should only use the training set for training your model and you should not use the feedback provided on the validation set to enhance the training. The restriction on the number of submissions limits this risk of "learning" on the validation set. Second, this limits the load on our servers.

Q. Can I use the validation set to tune the parameters of my model?

A. As explained in the previous two questions, the validation set is more like a test set for fun and it should not be used for extensive parameters selection. Instead we encourage you to do cross-validation on the training set to choose the best parameters for your model.

Q. Why use ERR instead of NDCG as an evaluation metric?

A. A drawback of NDCG is that its underlying user model is not realistic. For instance, a document coming after a very relevant document should be more discounted because it is likely that the user is already satisfied by the first document. NDCG fails to capture that while ERR does. Empirically we have found that ERR correlates better with user satisfaction than NDCG
(see Expected Reciprocal Rank for Graded Relevance).

But we are of course aware that NDCG is a widely used metric for ranking and this is why we also provide in the leaderboard results for NDCG. So if your algorithm is specifically tuned to optimize NDCG, you can compare yourself to others based on that metric.

Q. Why should I choose one of my submissions as my "primary entry?" Won't Yahoo! just do that for me?

A. First of all, even though you are allowed multiple submissions, you need to select a single one to be treated as your primary "entry" for the final evaluation. If all submissions were to be evaluated, that would give an unfair advantage to teams with a large number of submissions. If you do not designate a submission as your "primary entry", we will automatically select the one with the best score on the validation set. This might be a reasonable choice, but you often have some prior knowledge (or gut feeling!) to make a more informed decision. We thus encourage you to select yourself the primary submission.

Q. The data sharing agreement states that the data needs to be deleted by June 30th. How can it then be used in the future as a benchmark?

A. We will release the datasets under the Webscope program at the end of the competition. This release will include the relevance labels from the validation and test sets.

Q. Why not releasing the actual queries and urls?

A. For two reasons of competitive nature:

1. Feature engineering is a critical component of any learning to rank system. For this reason, search engine companies rarely disclose the features they use. Releasing the queries and urls would lead to a risk of reverse engineering of our features.
2. Our editorial judgments are a valuable asset, and along with queries and urls, it could be used to train a ranking model. We would thus give a competitive advantage to potential competitors by allowing them to use our editorial judgments to train their model.

Q. Can I compete even if I am not affiliated with an academic or industrial lab?

A. Yes, the competition is open to everyone. If you don't have an affiliation, just enter "self" while registering.

Q. Where can I find the Legal Rules?

A. To view the full Official Rules of the Contest click here.

Q. Who can I contact for other questions?

A. You can contact us at learningtorankchallenge@yahoo-inc.com or use the discussion forum of our Facebook group.