Course Overview
A hands-on graduate course on what happens after pretraining. Each week pairs a lecture on supervised fine-tuning, preference optimization, and reinforcement learning with a lab that puts the ideas into practice.
| Instructor | Chan Young Park · park@utexas.edu · UTA 5.442 |
|---|---|
| Office hours | Fridays 10–11am. Please email in advance to confirm. |
| Teaching Assistant | TBA |
| Class meetings | Wednesdays, 12–3pm · UTA 1.208 |
| Modality | In person. Attendance is expected; sessions are not livestreamed. |
| Canvas | utexas.instructure.com/courses/1454896 |
| Unique number | 30845 |
Course Description
A pretrained language model is not yet a useful assistant. Much of what makes modern LLMs useful happens after pretraining, including supervised fine-tuning, parameter-efficient adaptation, reward modeling, reinforcement learning from human and AI feedback, direct preference optimization, and reinforcement learning with verifiable rewards.
This is a hands-on course. Over the course of the semester, you will fine-tune open-weight models on GPUs, build evaluation suites for your own tasks, construct preference data, and train models with DPO and GRPO. We will use lecture time to develop the methods and the research behind them, and the labs are where we run the models, inspect the results, and see what breaks.
How This Course Works
Most Wednesdays are divided between lecture and a hands-on lab or workshop. The lecture introduces the methods, concepts, and research behind that week's topic. The lab or workshop provides time either to apply those ideas directly or to connect them to the group project through experimentation and discussion. A major component of the course is a group project: teams of three post-train a model for a task of their choosing, making decisions about data, evaluation, training methods, and iteration along the way.
Prerequisites
Required: Graduate standing; proficiency in Python; familiarity with basic machine learning concepts; and comfort with the command line. Prior coursework or equivalent experience in machine learning or NLP is expected.
Helpful but not required: Prior experience with PyTorch, the Hugging Face ecosystem, or HPC clusters. We will introduce the tools needed for the course, but students should be comfortable learning new ML libraries and working in a computational environment.
Learning Outcomes
| By the end of the course, students will be able to… | Assessed by |
|---|---|
| Decide whether a given task calls for prompting, retrieval, or post-training, and justify the decision with evidence | Proposal, concept check, final report |
| Design and implement a task-specific evaluation suite, including an LLM-as-a-judge component, and assess its validity | Evaluation labs, project milestones, final report |
| Implement supervised fine-tuning and parameter-efficient fine-tuning (LoRA/QLoRA) on open-weight models | HW1, project |
| Construct preference data and train a model with DPO; explain the relationship between RLHF and DPO | HW2, concept check |
| Understand reward modeling, PPO, RLAIF, and GRPO at a level sufficient to read current papers and choose among methods | Concept check, in-class lab |
| Diagnose training runs and common failure modes | HW1, HW2, final report |
| Communicate methods, results, and limitations to a technical audience | Final presentation, final report |
Course Materials
There is no required textbook. Readings will consist of research papers, documentation, and lab notebooks made available through Canvas at no cost. Papers discussed in the course may include:
- DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025
- Rafailov et al., “Direct Preference Optimization,” NeurIPS 2023
- Ouyang et al., “Training language models to follow instructions with human feedback,” NeurIPS 2022
- Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR 2022
- Bai et al., “Constitutional AI: Harmlessness from AI Feedback,” Anthropic, 2022
Required technology
- A laptop capable of SSH access and running a modern browser. Model training takes place on remote GPUs, not on your laptop.
- A TACC (Texas Advanced Computing Center) account, set up at the beginning of the course. Compute for coursework is provided through a class allocation at no cost.
- Access to course-provided remote model services for tasks such as model comparison, LLM-as-a-judge evaluation, and synthetic data generation. Onboarding instructions provided by the instructor.
If you run into problems with course infrastructure, including TACC, SLURM, or the course-provided remote model services, ask for help through Canvas or during TA office hours.