Leaderboard
The only requirement to appear on the leaderboard is to have a successful Run.
Being on the leaderboard makes you eligible for rewards based on the rewards system of the crunch you are participating in.
Submission Phase
During the Submission Phase, the leaderboard is updated when a Run is scored.
If there is no public leaderboard, the leaderboard will be updated when a Prediction is validated to make sure it is ready for scoring.
Out-of-Sample Phase
During the Out-of-Sample Phase, the leaderboard is only updated once when all the Run that are running on unseen data are scored.
Columns
Only a subset of columns will be visible, depending on the current state of the crunch.
Rank
Rank of the participant / model
Participant
Name of the participant / model
Participation
Distinguish if a participant is ready for the private leaderboard (only if there is no public leaderboard)
Team
The participant's team name (only if the competition allows teams)
Best metric
(unit
)
The best submission score for the metric metric
(reset between each rounds)
Last metric
(unit
)
The last submission score for the metric metric
Mean
The weighted average of the metrics (only if there are multiple)
metric
(unit
)
The score value of the metric metric
Run Success
The number of successful runs / the number of failed runs
Previous LB Chn
Rank change from the previous round
Public LB Chn
Rank change from the public leaderboard
Weekly Change
Rank change from the previous week
User Hist. Rewards
The amount of rewards ever received by the participant
Proj. Rewards
The amount of rewards the participant is expected to receive at the end of the month
Badges
The participant has won the crunch
The participant has earned a certificate
The participant has received a prize
The participant only submitted a Quickstarter and is disqualified from receiving any rewards
The code is not deterministic (running the infer function twice give a different result)
The score is not within the allowed range
Participant
If the crunch allows more than one model, the model name is displayed right after the participant name. Otherwise, only the participant name is displayed.
Only one Model on the Leaderboard
Some competitions (like Mid+One) allow more than one model, but only ONE can be displayed on the leaderboard at any given time.
Ties
Some competitions (such as Broad #2) allow ties when models achieve the same scores, meaning they receive the same rank.
The rank assigned to all tied models is the first rank of the tie.
The reward is distributed equally among the tied models, calculated as the sum of the rewards for the tied models divided by the number of models in the tie.
Example
For a prize pool that rewards the top 10 models:
If the 1st, 2nd, and 3rd place models are tied, they will all be ranked 1st. The next model will be ranked 4th. Their total reward will be: (1st place prize + 2nd place prize + 3rd place prize) / 3.
If the 8th, 9th, 10th, 11th and 12th place models are tied, they will all be ranked 8th. The next model will be ranked 13th. Their total reward will be: (8th place prize + 9th place prize + 10th place prize) / 5.
Last updated