Language Models

📄 Language Models

🎯 Learning objectives

• Understand how language models work
• Know the difference between traditional programming and language models (see the previous section for the foundation)
• Understand what a context window is and why it matters
• Know the most important limitations

In the previous section we talked about generative AI – now we zoom in on the most discussed type: language models (e.g. ChatGPT, Claude, Gemini). How do they work – and what do you need to know to use them effectively?

What is a language model?

Simply put: a language model is trained on an enormous amount of text. It predicts which word is most likely to come next, based on all the words that came before – patterns it learned from training data.

When you write “What is the capital of” the model leans toward the next word being a country name, followed by a response format with a city.

Modern models can answer complex questions, write code, summarize, translate, and reason – but they’re still generating text based on probability and patterns, not human “understanding” in the full sense.

How is a language model trained?

1. Data collection – Enormous amounts of text from books, articles, the web, Wikipedia, forums, and more.

2. Training – The model learns patterns through neural networks: which words follow each other, how sentences are structured, how different text types differ. Requires massive computation; millions of parameters are adjusted.

3. Fine-tuning – Humans rate responses so the model becomes more helpful, relevant, accurate, and safe.

Result: a model that can generate fluent text on almost any topic – within the scope of its training and knowledge cutoff.

What is a context window?

The context window is the total amount of text the model can work with at once – like short-term memory.

Everything must fit there: your question, previous messages, attached text, and the model’s own responses. It’s often measured in tokens (small text pieces – words, syllables, or characters).

Why does it matter?

1. Long documents – An entire book rarely fits; you must split it or use tools like RAG.
2. Long conversations – When the window is full the model drops the oldest parts (in Intric you may get an error if the context is full).
3. Context quality – More relevant context in the right order usually gives better answers.

RAG – giving the model current and comprehensive knowledge

RAG (Retrieval-Augmented Generation) lets the model avoid cramming an entire giant knowledge base into a prompt every time.

Instead:

1. Documents are split into smaller chunks with metadata (document, page, section).
2. When you ask a question, the system picks the chunks that seem most relevant.
3. Only those pieces are sent into the context window together with your question.

It’s like a librarian who fetches the right chapter for you instead of dumping the whole shelf on the table.

Limitations of language models

1. Knowledge cutoff – Training ends at a certain date; events after that don’t exist in the “base model” (connections to the web are a separate layer).

2. Hallucinations – The model can sound convincing but be wrong – especially when filling gaps without adequate sources.

3. No real understanding – Strong on text patterns; weak on anything requiring real experience, sensation, and shared worldview.

4. Context limit – You’re constrained by the context window size and how well the right information actually gets included.

5. Inconsistency – The same question can give slightly different answers (sampling, temperature, small wording differences).

Summary

• Language models predict the next word based on patterns in training data.
• The context window limits how much can be included at once.
• RAG fetches relevant excerpts so you don’t have to fill entire documents into the prompt.
• Key risks: knowledge cutoff, hallucinations, lack of “real” understanding, and variation in responses.

Test your knowledge

3 questions · 100% correct to pass · Review your answers when done

Start quiz →