<chat-markup-language>
The ChatML Handbook
A Developer's Guide to Structured Prompting and LLM Conversations
Structure turns prompting from guesswork into engineering: when conversations are expressed as explicit, role-tagged, validated messages, AI behavior becomes predictable, traceable, reproducible, and safe to operate in production.
The question this book opens with
Is ChatML still relevant?
A fair question before you invest: ChatML began as an OpenAI markup in 2023, and OpenAI's own public API barely names it today. So is this a dead format? No - and the honest answer is the reason this book exists. The category won.
Structured, role-tagged messages with explicit boundaries are now the interchange
format of the entire open-model ecosystem: vLLM, Text Generation Inference, and
Hugging Face chat templates all speak an ordered list of role-tagged messages, and
current models like Qwen and SmolLM2 ship the literal
<|im_start|> / <|im_end|>
tags this book uses.
What did not happen is a single universal markup. Providers chose different tags - Claude uses an XML-style dialect of its own - so the field landed on a category that won and a hundred dialects that never converged. That is exactly why this book teaches the durable grammar (roles, boundaries, structured turns) rather than any one vendor's tag set.
The role-tagged message list won as the universal format; the specific tokens fragmented. Learn the grammar, not the tags.
<|im_start|>system
You are a precise, structured assistant.
<|im_end|>
<|im_start|>user
What is ChatML?
<|im_end|>
<|im_start|>assistant
A markup that gives every message a
role, a boundary, and an order.
<|im_end|>- system
- user
- assistant
- boundaries
The canonical wire form used throughout the book
The centerpiece
Eleven chapters, and the position each one takes
Open a chapter to see its argument, the artifact it leaves in your codebase, and the position it commits to. The positions are the point: this is a book that decides, rather than surveying options.
Part I — Foundations
Conceptual DNA: history, message anatomy, roles, context and memory, design principles.
Chapter 1 The Evolution of Structured Prompting argument · what you build · the position it takes
The argument
Structure emerged from necessity as conversational AI grew; ChatML is the standardized result.
What you build
- Message dataclass(role, content)
- render_chatml(messages) -> str
The position it takes
- Unstructured prompting does not scale to multi-turn, multi-role systems.
- Pseudo-role prefixes are injection-by-default.
- The category won, not the specific tags: a role-tagged message list is the ecosystem-level thin waist.
Chapter 2 Anatomy of a ChatML Message argument · what you build · the position it takes
The argument
The ChatML message is the atomic unit; boundaries plus role plus metadata make state reconstructable.
What you build
- Message dataclass(role, content, metadata)
- Message.to_dict() -> dict
The position it takes
- A message is to a conversation what a class is to code.
- Content is data; structure (boundary and role) is yours.
- The function_call shape is model-specific - validate it.
Chapter 3 Roles and Responsibilities argument · what you build · the position it takes
The argument
Roles enforce conversational integrity through authority boundaries; the system role is immutable.
What you build
- role_injection_guard(user_text) -> str
- _ROLE_MARKERS regex
- detect_order_id()
The position it takes
- Role isolation is a safety control, not a formatting nicety.
- Authority flows down the hierarchy, never up.
- Tool outputs are untrusted data, like user input.
Chapter 4 Context and Continuity argument · what you build · the position it takes
The argument
Stateless LLMs need explicit context chains; memory is replayed history, not model state. Bound the replay tax with a context budget.
What you build
- get_history(session_id)
- trim_to_window(history, max_turns)
- summarize_old_turns()
- estimate_prompt_tokens()
The position it takes
- Memory is an application concern, not a model feature.
- The naive send-everything loop is quadratic - the replay tax.
- A single JSON file is demo-only; multi-worker needs a locking store.
- Pick the context budget before the model.
Chapter 5 Design Principles of ChatML argument · what you build · the position it takes
The argument
Three pillars - structure, hierarchy, reproducibility - make dialogue machine-interpretable and human-verifiable.
What you build
- render_chain(chain) -> str
- context_fingerprint(chain) -> str (SHA-256)
The position it takes
- ChatML makes the input reproducible, not the model output.
- Reproducibility is engineered and logged, not granted.
- Use the name field and metadata, not invented tokens, for sub-agents and envelopes.
Part II — Engineering
Operationalize ChatML: pipeline, templates, tools, persistence, testing and observability.
Chapter 6 Building a ChatML Pipeline argument · what you build · the position it takes
The argument
A ChatML pipeline is three layers - input, logic, output - connected by the thin waist: one capsule schema. You do not parse ChatML out of generations.
What you build
- build_user_message
- assemble_context
- route(message, handlers)
- handle_request (end to end)
- log_turn
The position it takes
- Prompt design is software architecture.
- You do not string-parse ChatML out of a model generation - the runtime returns a structured turn.
- Model and tools are untrusted dependencies; the pipeline contains their failures.
- Constraints must live in code, not in the prompt.
Chapter 7 Rendering with Templates argument · what you build · the position it takes
The argument
Jinja2 templating makes dynamic ChatML generation reproducible and governable at scale.
What you build
- _env (cached Environment, StrictUndefined)
- render_capsule(role, template_name, context)
- render_checked
- templates/*.jinja2 per intent
The position it takes
- Templates are policy artifacts - version them.
- User content is a context variable, never template source (SSTI and RCE).
- StrictUndefined: missing variables must fail loud.
- One cached Environment, not one per request.
Chapter 8 Tool Invocation and Function Binding argument · what you build · the position it takes
The argument
Tools turn reasoning into action; a typed registry, a sandbox, and observability make it safe.
What you build
- register_tool(name, schema, writes=)
- execute(name, raw_args, idempotency_key)
- _run_with_retry (backoff)
- run_bounded (ThreadPool timeout)
- allow() rate limiter
The position it takes
- Tool intents are untrusted - the model picks the name and args, so whitelist and validate with Pydantic.
- The side-effect boundary: reads retry, writes need idempotency keys and run once.
- A timed-out write is why idempotency exists.
- Errors become capsules, never raise to the model.
Chapter 9 Memory Persistence Layer argument · what you build · the position it takes
The argument
Durable memory equals embeddings plus a vector store plus hybrid retrieval, with privacy as a first-class concern.
What you build
- embed_message(content)
- recency_weight (exponential decay)
- hybrid_search (over-fetch and re-rank)
- mask_pii (regex)
- consolidate / replay_context
The position it takes
- Recency must be a computed decay blended with similarity, not a magic constant.
- Embedding cohort: vectors are only comparable within one model version; an upgrade means re-embedding.
- Memory is a privacy liability: mask PII, enforce retention, keep writes idempotent.
Chapter 10 Testing and Observability argument · what you build · the position it takes
The argument
Determinism is testable: structural validation, reproducibility checksums, and regression eval build trust.
What you build
- validate_capsule / CAPSULE_SCHEMA
- assert_valid_sequence
- BERTScore semantic test with threshold
- OTel traced_model_call span
- record() Prometheus counter
The position it takes
- The assertion line: exact tests for structure and context, tolerant semantic tests for generation.
- Never assert that model output equals a fixed string.
- SLOs turn the earlier chapters' failure modes into pagers.
Part III — The Support Bot Project
Capstone: a full production support bot, with the appendices as reference.
Chapter 11 Building a Support Bot Using ChatML argument · what you build · the position it takes
The argument
All prior layers compose into one production-grade, deployable support bot.
What you build
- FastAPI /chat endpoint
- ChatQuery Pydantic model
- tool registry
- memory manager
- structured logging
- Dockerfile
The position it takes
- Structure is what separates a deployable bot from a demo.
- Every layer is load-bearing - deleting one reproduces a specific failure.
- The model is the smallest part of a reliable system.
- The integration contract: the next feature is wiring, not rewriting.
The vocabulary the book leaves you with
One named concept per chapter
A note titled "Named concept" introduces one of the book's coined terms - a pattern you can name in your own team afterward. They are collected in Appendix D.
-
Conversational grammar
The set of rules that say which structural pieces a dialogue is made of and how they may be arranged - roles, boundaries, and ordering.
Chapter 1 -
Conversation capsule
One ChatML message treated as an indivisible unit: a role, its content, and any metadata, sealed between the start and end markers.
Chapter 2 -
The constitutional layer
The system message treated as immutable, application-owned law: set by your code, never by user input, and outranking everything below it.
Chapter 3 -
The replay tax
Each new turn costs you the tokens, latency, and money of every turn that came before it. Over a full conversation the token cost is quadratic, not linear.
Chapter 4 -
The context envelope
A logically grouped set of messages relevant to one objective, tagged so they can be retrieved or retired together.
Chapter 5 -
The ChatML thin waist
The single message schema - a role, content, and metadata capsule - that every pipeline layer speaks.
Chapter 6 -
Template provenance
The record, stamped on every rendered capsule, of which template - and which version - produced it.
Chapter 7 -
The side-effect boundary
The line between tools that only read and tools that change the world. Every reliability technique is safe before the boundary and dangerous after it.
Chapter 8 -
The embedding cohort
The set of vectors produced by one specific embedding model version. Vectors are only comparable within a cohort.
Chapter 9 -
The assertion line
Separates what is deterministic and yours from what is probabilistic and the model's. Almost every flaky LLM test crosses it.
Chapter 10 -
The integration contract
A codebase has an integration contract when its next feature is a wiring diagram, not a refactor.
Chapter 11
What you run while you read
A deliberately small, local-first stack
The examples target a stack you can run on a laptop with no API bill. The architecture is provider-agnostic: swap the model call for a hosted endpoint and only the wire dialect changes.
- 0API bill
- 1.5Bparameter local model
- ~1 GBone-time model download
- 3environment variables
The stack
| Component | Version / choice |
|---|---|
| Language | Python 3.10+ |
| Web framework | FastAPI |
| Model runtime | Ollama |
| Model | Qwen2.5:1.5B |
| Templating | Jinja2 |
| Validation | Pydantic |
| Vector store (Chapter 9) | Qdrant |
Every variable has a default
| Variable | Default |
|---|---|
LLM_API | http://localhost:11434/api/chat |
LLM_MODEL | qwen2.5:1.5b |
CHATML_RECENCY_TURNS | 10 |
You only set them to override.
Readership
Who this book is for
- AI developers and engineers building production-grade conversational systems.
- Researchers exploring reasoning, orchestration, and multi-agent collaboration.
- Educators and students seeking a conceptual framework for dialogue systems.
- Product and platform teams integrating LLMs with APIs, memory, and tools.
Whether you are experimenting with prompt templates or architecting large-scale agentic systems, this handbook offers both a conceptual compass and a practical toolkit.
From the back cover
What it covers
The ChatML (Chat Markup Language) Handbook is a practical guide for developers, AI engineers, and technical creators who want to build reliable, structured, and controllable interactions with large language models. It explains how ChatML organizes conversations using roles, hierarchy, and message boundaries - turning chaotic prompts into well-defined conversational logic - and walks through modern prompting, multi-turn design, function calling, and tool integration.
Beyond theory, it delivers hands-on frameworks, real-world examples, and complete workflows for designing AI assistants, multi-agent systems, and ChatML-compliant pipelines. You will learn to debug conversations, enforce system policies, build tools and agents, and create scalable pipelines - equipping you with the discipline to design intelligent, predictable, production-grade conversational systems.
Table of contents
Three parts, then the reference shelf
Part I — Foundations
- 01 The Evolution of Structured Prompting
- 02 Anatomy of a ChatML Message
- 03 Roles and Responsibilities
- 04 Context and Continuity
- 05 Design Principles of ChatML
Part II — Engineering
- 06 Building a ChatML Pipeline
- 07 Rendering with Templates
- 08 Tool Invocation and Function Binding
- 09 Memory Persistence Layer
- 10 Testing and Observability
Part III — The Support Bot Project
- 11 Building a Support Bot Using ChatML
Front & back matter
- — Conventions Used in This Book
- — Using the Code Examples
- — Epilogue: Beyond ChatML
- — Further Reading
Part IV — Appendices (Ecosystem & Reference)
- A ChatML Syntax Reference
- B Integration Ecosystem
- C Template & Snippet Library
- D Glossary & Design Checklist
- E ChatML and Today's Provider APIs
Companion code
Anchored in one runnable system
The book is anchored in one runnable system, the ACME Support Bot v3.4. The chapters excerpt and explain its code; the complete, runnable project lives in a public repository.
Code builds progressively across chapters: a function introduced once is reused later by name rather than re-implemented. A grey bar above each block names the file the code belongs to.
Running it
git clone https://github.com/ranjankumar-gh/support-bot-v3.4
cd support-bot-v3.4
pip install -r requirements.txt
ollama pull qwen2.5:1.5b # one-time model download
uvicorn app:app --reload # starts the /chat APIThe README in the repository carries the authoritative setup steps and any version updates.