Announcing release v5.0.0
Here are the main changes:
Changes
1. Native support for multi-dataset evaluation
Immediate effect
Implemented support for multi-dataset evaluation for all competitions using EvalTasks from taoverse 1.3.7
2. Updating the data mix
Activation block: 4_732_978
Both 3B and 14B will now be evaluated on the following mix:
- HuggingFaceFW/fineweb (30%)
- HuggingFaceFW/fineweb-edu-score-2 (25%)
- bigcode/the-stack-v2-dedup (35%)
- laion/pes2ox-fulltext (5%)
- HuggingFaceTB/FineMath:finemath-3plus (3%)
- HuggingFaceTB/FineMath:infiwebmath-3plus (2%)
Notice that the 14B* will be retired and replaced by the multi-dataset 14B with the above mix.
3. New epsilon lower bounds and decay intervals
Activation block 4_732_978
The epsilon decay interval and bounds will be updated for all competitions as follows:
-
3B competition:
Updated: decays from 0.005 to 0.0002 over 4 days
Updated: decays from 0.005 to 0.0005 over 7 days -
14B competitions:
Updated: decays from 0.005 to 0.0002 over 5 days
Updated: decays from 0.005 to 0.0005 over 10 days
4. Updated emission distribution for competitions
Activation block 4_732_978
- 3B → 30%
- 14B → 70%
Other updates
- Improved weight setting by spinning out separate threads that use different subtensors.
- Switched from bt.logging to taoverse.logging.
NOTES TO VALIDATORS
-Please also make sure to rerun pip install to ensure updated dependencies.
python -m pip install -e