FAQ

1. What is the difference between the Sandbox and the Annual Competition?

The Sandbox is for ongoing public evaluation using the Sandbox data only. Scores and rankings are updated continuously so participants can test and improve their models.

The Annual Competition is a holdout challenge. Participants may use the Sandbox training data, the Sandbox test data, or any other data source available to them to develop their models, but winners are determined only from predictions on the Annual Competition test input file. Final winners are selected using the average performance across the 30, 60, 90, and 120 minute prediction horizons for the two primary metrics: MARD and DTS Error Grid Zone A.

2. What data can be used to build the model?

A model may use any data beyond the MetaboNet data in any way, i.e., with or without forward-looking input (FLI). However, the final official evaluation for this year's annual competition will only have FLI of insulin and meal information. In general, information on timing, intensity, and duration of exercise may also be used as FLI, if available in future annual competition datasets, while data such as heart rate and CGM readings should never be used because there is no way to collect them ahead of time in practice.

3. Is the final ranking based on the average of time horizons for a given metric, or a single time horizon?

It is based on the average error across the 30, 60, 90, and 120 minute time horizons for each primary metric: MARD and DTS Error Grid Zone A.

4. Why was Live Leaderboard renamed to Sandbox?

The "Sandbox", what was previously called "Live Leaderboard", is designed to help participants familiarize with the submission process of annual competition. The test data for "Sandbox" is fully available, thus, the ranking is less meaningful unless a model has been open-sourced and officially validated. In contrast, the ranking of "Annual Competition Leaderboard" will be officially released after evaluation is done on the held-out test. For details, please refer to "Ranking Metrics and Winners".

5. What is included in the column "included_in_eval" of the file "annual_competition_secret_holdout_set_public.parquet" downloadable in https://metabo-net.org/glucose-prediction-challenge?

We would like to clarify that the secret holdout set may overlap with existing public datasets. To prevent data leakage between publicly available datasets and the secret hold-out set, we designated a segment in the hold out set as a washout period which will be excluded from the final evaluation if the value for "included_in_eval" is false.