How to Learn AI by Building LLM Apps

Learn AI by shipping small LLM apps

The fastest way to learn AI as a developer is to build useful LLM applications, measure how they behave, and improve them with real data. Courses can help with vocabulary, but they will not teach you how prompts fail in production, how model latency affects a user flow, or how one prompt edit can break JSON output.

A good learning project should force you to handle the same issues you will see at work: prompt design, model calls, structured outputs, evals, logging, versioning, privacy, and deployment. Start small. Build one workflow that solves one narrow problem, then make it reliable.

Pick a small app with a testable outcome

Do not start with a general agent that can “do anything.” Start with an LLM feature where you can tell whether the output is good or bad.

Good first LLM app ideas

• Support ticket classifier: route tickets into billing, bug, account, or sales.
• Meeting note cleaner: turn rough notes into action items and decisions.
• PR review assistant: flag missing tests, risky changes, and unclear code comments.
• Internal docs Q&A: answer questions using a small folder of approved docs.
• Email reply drafter: produce a draft with tone and policy constraints.

For this tutorial, use a support ticket classifier. It has clear inputs, clear labels, and easy evals.

Basic app architecture

Your first LLM app does not need a complex stack. It needs a clean path for inputs, prompts, model calls, logs, evals, and prompt versions.

User or test case
      |
      v
Web app or API route
      |
      v
Input validation and PII checks
      |
      v
Prompt template + prompt version
      |
      v
LLM provider API
      |
      v
Output parser and schema validation
      |
      v
Application response
      |
      +----------------------------+
                                   |
                                   v
                         Logs, traces, latency,
                         cost, prompt version,
                         model, input, output
                                   |
                                   v
                         Eval dataset and reports

This structure teaches the real workflow: build, log, evaluate, change one thing, evaluate again. If you skip the logging and eval pieces, you are mostly guessing.

Step 1: Make your first API call

Start with one direct model call. Do not add retrieval, tools, memory, or agents yet. You want to understand the raw behavior of the model first.

Example: first API call with the OpenAI Responses API

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});

const ticket = `
I was charged twice this month. I already paid my invoice last week.
Can someone fix this?
`;

const response = await client.responses.create({
  model: "gpt-4.1-mini",
  input: [
    {
      role: "system",
      content: "You classify support tickets. Return only valid JSON."
    },
    {
      role: "user",
      content: `Classify this ticket into one category: billing, bug, account, sales, other.

Ticket:
${ticket}

Return JSON with this shape:
{
  "category": "billing | bug | account | sales | other",
  "confidence": 0.0,
  "reason": "short explanation"
}`
    }
  ]
});

console.log(response.output_text);

Example output

{
  "category": "billing",
  "confidence": 0.94,
  "reason": "The user reports being charged twice and asks for a billing correction."
}

This tiny example already gives you several learning targets:

• How system and user messages affect behavior.
• How to ask for structured output.
• How often the model returns valid JSON.
• How confident the model is on easy and ambiguous tickets.
• How latency and cost change by model.

Step 2: Turn the prompt into a versioned asset

Many teams keep prompts as random strings in application code. That works for a demo, then breaks down when you need to compare versions, roll back a bad prompt, or explain why an output changed.

Track each prompt version with at least these fields:

• Prompt name: support-ticket-classifier
• Version: v1.0.0
• Model: gpt-4.1-mini
• Inputs: ticket_text
• Output schema: category, confidence, reason
• Owner: team or developer responsible
• Change note: what changed and why

Example prompt version record

{
  "name": "support-ticket-classifier",
  "version": "1.0.0",
  "model": "gpt-4.1-mini",
  "temperature": 0,
  "input_variables": ["ticket_text"],
  "output_schema": {
    "category": ["billing", "bug", "account", "sales", "other"],
    "confidence": "number between 0 and 1",
    "reason": "string under 160 characters"
  },
  "change_note": "Initial classifier for five support categories."
}

Example prompt template

System:
You classify support tickets. Return only valid JSON.
Do not include markdown. Do not include extra keys.

User:
Classify this ticket into one category:
- billing
- bug
- account
- sales
- other

Ticket:
{{ticket_text}}

Return JSON:
{
  "category": "billing | bug | account | sales | other",
  "confidence": 0.0,
  "reason": "short explanation"
}

Once you version prompts, every production request should record the prompt name and version. That gives you a way to connect user issues, eval failures, and model behavior to the exact prompt that produced the output.

Step 3: Log every request while you build

Learning AI without logs is slow. You need to see inputs, outputs, prompt versions, latency, cost, errors, and parser failures. This is the practical side of LLM observability.

At minimum, log this for each request:

• Request ID
• User or account ID, if allowed by your privacy rules
• Prompt name and version
• Model name
• Temperature and key parameters
• Input length
• Raw model output
• Parsed output
• Latency in milliseconds
• Estimated cost
• Error type, if any

Example trace record

{
  "request_id": "req_2026_06_02_0019",
  "prompt_name": "support-ticket-classifier",
  "prompt_version": "1.0.0",
  "model": "gpt-4.1-mini",
  "temperature": 0,
  "latency_ms": 842,
  "input_tokens": 118,
  "output_tokens": 42,
  "parsed": true,
  "output": {
    "category": "billing",
    "confidence": 0.94,
    "reason": "The user reports a duplicate charge."
  }
}

Do not wait until production to add this. Logs are how you learn which failures are real. They also prevent a common mistake: optimizing prompts based on a few hand-picked examples in a chat UI.

Step 4: Build a small eval dataset

An eval dataset is a set of inputs with expected behavior. For the support classifier, start with 50 tickets. Keep them simple at first, then add edge cases.

Example eval dataset

[
  {
    "id": "ticket_001",
    "ticket_text": "I was charged twice for my subscription this month.",
    "expected_category": "billing"
  },
  {
    "id": "ticket_002",
    "ticket_text": "The export button returns a 500 error every time.",
    "expected_category": "bug"
  },
  {
    "id": "ticket_003",
    "ticket_text": "Can you tell me pricing for 200 seats?",
    "expected_category": "sales"
  },
  {
    "id": "ticket_004",
    "ticket_text": "I cannot reset my password because the email never arrives.",
    "expected_category": "account"
  }
]

Start with exact-match scoring for classification. If the model returns the correct category, it passes. If it returns the wrong category, it fails. This keeps the feedback loop clear.

As your app gets more complex, use richer LLM evaluation methods. For free-form answers, you may need rubric grading, reference checks, retrieval checks, or LLM-as-a-judge scoring.

Step 5: Run evals before changing prompts

A prompt edit should have a before-and-after report. If you change the prompt and only read five outputs manually, you will miss regressions.

Example eval runner

import fs from "node:fs/promises";

const dataset = JSON.parse(
  await fs.readFile("./evals/support_tickets.json", "utf8")
);

let passed = 0;
const results = [];

for (const testCase of dataset) {
  const output = await classifyTicket(testCase.ticket_text);

  const pass = output.category === testCase.expected_category;

  if (pass) passed += 1;

  results.push({
    id: testCase.id,
    expected: testCase.expected_category,
    actual: output.category,
    confidence: output.confidence,
    pass
  });
}

const passRate = passed / dataset.length;

console.table(results);
console.log(`Pass rate: ${(passRate * 100).toFixed(1)}%`);

Example eval results

In this example, version 1.1.0 is probably better. It improved classification accuracy and fixed JSON parsing. The latency and cost increase is small enough for most support workflows. In a high-volume workflow, you might set stricter cost limits.

Step 6: Add schema validation

LLM output should not move directly into your product logic. Parse it, validate it, and handle failures.

Example validation with Zod

import { z } from "zod";

const ClassificationSchema = z.object({
  category: z.enum(["billing", "bug", "account", "sales", "other"]),
  confidence: z.number().min(0).max(1),
  reason: z.string().max(160)
});

function parseClassification(rawText) {
  let parsed;

  try {
    parsed = JSON.parse(rawText);
  } catch {
    return {
      ok: false,
      error: "invalid_json"
    };
  }

  const result = ClassificationSchema.safeParse(parsed);

  if (!result.success) {
    return {
      ok: false,
      error: "schema_validation_failed",
      details: result.error.flatten()
    };
  }

  return {
    ok: true,
    data: result.data
  };
}

This teaches an important production habit: treat the model as an unreliable dependency. Even strong models can return malformed output, omit fields, or follow the wrong instruction when the input is messy.

Step 7: Add privacy checks early

Many AI learning projects use real customer data too casually. Do not copy production tickets into a notebook or third-party tool without checking your data policy.

For a learning app, use one of these approaches:

• Use synthetic examples that match real formats.
• Redact names, emails, phone numbers, API keys, addresses, and payment details.
• Store only the fields you need for debugging and evals.
• Set retention rules for logs.
• Check whether your provider uses data for training by default.

Example PII redaction helper

function redactPII(text) {
  return text
    .replace(/[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi, "[redacted_email]")
    .replace(/\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b/g, "[redacted_phone]")
    .replace(/\b(?:\d[ -]*?){13,16}\b/g, "[redacted_card]");
}

const safeTicketText = redactPII(ticketText);

Privacy is easier to handle when it is part of the architecture. Retrofitting it later is painful, especially after logs contain sensitive data.

Step 8: Improve the prompt with measured changes

Once you have logs and evals, improve the prompt one change at a time. Change too many things at once and you will not know what caused the result.

Example prompt change

Problem: Tickets about failed password reset emails are often classified as bugs instead of account issues.

Prompt version 1.1.0 change:

Add this rule:
If the user cannot log in, reset a password, verify an email, update a profile,
or access an existing account, classify the ticket as "account" unless the ticket
clearly reports a product error code or broken feature.

Expected eval impact:

• Account category pass rate improves.
• Bug category pass rate does not drop by more than 2%.
• JSON parse rate stays at 100%.
• Average latency stays under 1,000 ms.

This is the build-measure-iterate workflow. It turns prompt work into engineering work.

Step 9: Add retrieval only when the app needs facts

Some LLM apps need external context. A support classifier usually does not. A docs Q&A app does.

Add retrieval when the model must answer using a changing knowledge base, such as:

• Product documentation
• Internal policies
• Customer-specific configuration
• Release notes
• Pricing rules

When you add retrieval, your evals need more checks:

• Did the retriever fetch the right document?
• Did the answer use the retrieved context?
• Did the model invent facts outside the context?
• Did the answer cite the right source?

This is where many teams confuse model quality with retrieval quality. If the model gives a bad answer because the wrong document was retrieved, a prompt rewrite may not fix it.

Step 10: Use agents later, after the workflow is stable

Agents are useful when a system needs to plan, call tools, inspect results, and continue. They also add more failure modes. If you use agents too early, you may spend your time debugging tool loops instead of learning core LLM application behavior.

Before adding an agent, ask:

• Can this be solved with one model call?
• Can this be solved with a fixed chain of two or three calls?
• Do we need dynamic tool choice?
• Do we have evals for each step?
• Do we log every tool call and intermediate output?

For the support ticket app, an agent is probably unnecessary at first. A better next step is a fixed workflow:

1. Classify the ticket.
2. Extract priority and sentiment.
3. Draft a reply if confidence is above 0.85.
4. Send low-confidence cases to a human reviewer queue.

This keeps the workflow understandable and easier to evaluate.

A practical 30-day learning plan

You can learn a lot in 30 days if you build one app and improve it with real measurements.

Days 1 to 3: Build the smallest working version

• Pick one workflow.
• Make one model call.
• Return structured JSON.
• Validate the output schema.
• Save 20 example inputs.

Days 4 to 7: Add prompt versions and logs

• Create a prompt template.
• Track prompt version, model, latency, input tokens, and output tokens.
• Log parser failures.
• Write down the first three failure patterns you see.

Days 8 to 14: Build evals

• Create a 50-case eval dataset.
• Run exact-match scoring for classification.
• Track pass rate, parse rate, latency, and cost.
• Test at least two prompt versions.

Days 15 to 21: Improve reliability

• Add better instructions for common failures.
• Add retry behavior for invalid JSON.
• Add privacy redaction.
• Add error handling for provider timeouts.
• Set minimum quality gates, such as 90% pass rate and 99% parse rate.

Days 22 to 30: Ship a limited version

• Deploy the app behind an internal route.
• Collect logs for real usage.
• Add 20 real, redacted failures to the eval dataset.
• Compare new prompt versions before rollout.
• Document what changed and what still fails.

Common mistakes to avoid

Only watching courses

Courses teach concepts. Building teaches constraints. You need both, but the learning compounds when you ship small features and inspect failures.

Skipping evals

If you do not run evals, every prompt change is a guess. Even a small instruction edit can improve one case and break ten others.

Using agents too early

Agents add planning, tool selection, state, and retries. Learn single-call and fixed-chain workflows first. Add agents when the problem requires dynamic decisions.

Ignoring data privacy

Do not treat logs as harmless. Prompts often contain customer messages, internal documents, secrets, or personal data. Redact early and set retention rules.

Optimizing prompts without logs

You cannot improve what you cannot inspect. Logs show the real distribution of inputs, common parser failures, latency spikes, and bad outputs.

Treating AI learning as pure theory

LLM application work is an engineering loop: build, measure, change, measure again. The theory matters more when you connect it to working software.

What you should know after building one LLM app

After this project, you should understand:

• How prompts control model behavior.
• How structured outputs fail and how to validate them.
• How to track prompt versions.
• How to build a small eval dataset.
• How to compare prompt and model changes.
• How logs and traces make debugging possible.
• When retrieval helps and when it adds noise.
• Why agents should come after reliable smaller workflows.

That is a strong base for more advanced AI engineering work, including retrieval-augmented generation, tool calling, multi-step prompt chains, CI evals, and production monitoring.

Final checklist

• Build one narrow LLM app.
• Start with one direct API call.
• Return structured output.
• Validate the schema.
• Track prompt versions.
• Log every request during development.
• Create at least 50 eval cases.
• Run evals before and after prompt changes.
• Protect sensitive data.
• Add retrieval, tools, and agents only when the app needs them.

PromptLayer helps AI teams manage prompt versions, trace LLM requests, run evals, and compare changes before they reach production. If you are learning AI by building real LLM apps, create a PromptLayer account and start tracking your prompts and evals at https://dashboard.promptlayer.com/create-account.