skinny.

Latest / Misar.Blog Podcast

How to Prepare a Dataset for RLHF and DPO Fine-Tuning

Step-by-step guide to preparing preference datasets for RLHF and DPO: JSONL formats, human vs AI annotation, quality filters for length and position bias, pair… From the article "How to Prepare a Dataset for RLHF and DPO Fine-Tuning" by Synor, published on Misar.Blog.

In this episode:

0:00 A Preference Dataset for RLHF/DPO Is a JSONL File Of

2:05 Good Preference Data Should Come from Sampled Candidate Responses

3:53 The Article Recommends Different Pair Counts for Different Goals

5:57 A Preference Pair Is the Atomic Unit for DPO

7:13 Rejected Responses Should Be Plausible-But-Worse Rather…

The skinny

The skinny isn't ready yet — notes appear once the transcript is processed.