The 212AY Library
AI
Glossary.
200+ artificial intelligence terms explained in plain language — from machine learning and LLMs to AI agents, RAG and prompt engineering. Every definition comes with a concrete example, in English, French and Arabic.
200 terms
Base Model
A base model is a large language model that has been trained on vast text data to predict the next token but has not yet been fine-tuned for specific tasks or aligned with human preferences. It is raw talent: brilliant at completing text but not yet trained to follow instructions politely or refuse harmful requests. Base models serve as the starting point for creating chatbots and assistants through further training stages like instruction tuning and RLHF.
Batch Normalization
Batch normalization is a technique that standardizes the inputs to each layer of a neural network during training, keeping the data distribution stable as it flows through the network. This speeds up training dramatically, allows higher learning rates, and acts as a mild regularizer. Without it, deep networks are prone to internal covariate shift, where each layer must constantly adapt to changing input distributions, slowing convergence.
Batch Size
Batch size is the number of training examples processed together before updating model weights. Small batches use less memory and may generalize better but train slowly; large batches train faster with GPU parallelism but may overfit. Typical values range from 16 to 512. Choosing batch size balances training speed, memory constraints, and model quality.
Benchmark
A benchmark is a standardized test or dataset used to measure and compare the performance of AI models on specific tasks. Think of it as the final exam that every model takes under the same conditions. Common benchmarks include MMLU for general knowledge, HumanEval for coding, and GLUE for language understanding. They help researchers and businesses choose the right model, but benchmarks have limits: a model that aces a benchmark may still struggle with real-world messy data.
BERT
BERT (Bidirectional Encoder Representations from Transformers) is a foundational language model that reads text in both directions simultaneously, allowing deeper context understanding than left-to-right models. Released by Google in 2018, it kickstarted the modern pre-trained language model era and excels at classification, question answering, and named entity recognition. Its architecture influenced GPT, RoBERTa, and many successors.
Bias-Variance Tradeoff
The bias-variance tradeoff describes the balance a model must strike between being too simple (high bias, underfitting) and too complex (high variance, overfitting). A model with high bias misses real patterns, like a student who studies only summaries and fails nuanced questions. A model with high variance memorizes noise in training data and fails on new inputs, like a student who memorizes answers but cannot handle a differently worded exam. Good modeling finds the sweet spot.
BLEU Score
BLEU (Bilingual Evaluation Understudy) is an automatic metric measuring similarity between machine-generated text and human reference translations based on n-gram overlap, scoring from 0 to 1. Widely used for evaluating machine translation, BLEU has known limitations: it cannot capture meaning, rewards repetition, and penalizes valid alternative phrasings. It remains a useful baseline despite its flaws.
Byte-Pair Encoding (BPE)
BPE is a tokenization algorithm that starts with individual characters and iteratively merges the most frequent token pairs until a desired vocabulary size is reached. It balances word-level tokenization (huge vocabularies, no unknown words) and character-level (very long sequences). BPE is used by GPT, Llama, and most modern language models, handling English, French, Arabic, and code within a unified vocabulary.
Catastrophic Forgetting
Catastrophic forgetting occurs when a neural network loses its ability to perform a previously learned task after being fine-tuned on a new task, because new training overwrites the weights encoding original knowledge. This is a major challenge in continual learning. Techniques like elastic weight consolidation and replay buffers help mitigate it by preserving important weights during new training.