Chat G&T began as a practical question: could a small language model reliably turn any request into a useful answer expressed as a structured cocktail recipe?
The problem
The product needed to do several things at once. It had to answer the underlying request, express that answer through a coherent cocktail metaphor, produce a plausible recipe and follow a strict output schema. A long prompt with worked examples could describe that contract, but every request had to carry the same context again.
I wanted to find out which parts of the behaviour could be moved into a fine-tuned model without assuming that a more reliable format meant a better answer.
What I built
I built an interactive product around two configurations of the same 1.5-billion-parameter model:
- a detailed system prompt with five worked examples;
- a LoRA adapter trained for the task, using a minimal system prompt.
Both systems answered the same 60 prompts withheld from training. Each prompt was attempted once, with no repair, regeneration or manual correction.
The public site includes a “Spirit Guide” for trying the model, a blind “Tasting Room” for comparing outputs, the experimental method, held-out results and the underlying learning notes. The output contract covered schema validity, usefulness, metaphor, recipe quality, token use and latency.
Where reality disagreed
The fine-tuned system was more structurally reliable: it returned 52 schema-valid first attempts from 60 prompts, compared with 43 for the five-shot baseline. It also reduced average recurring input from approximately 2,576 tokens to 32.
But those gains did not settle the more important question. Human review did not consistently support the automated evaluator’s apparent improvement in answer quality, and average generation time increased from 3.92 to 5.70 seconds in the matched test.
The model learned the response contract more clearly than it improved the substance of the answers.
How I investigated it
I kept the base model, task and held-out prompt set fixed, then compared first attempts rather than polished outputs. Automated checks handled properties such as schema validity; model-assisted and human review examined usefulness, metaphor and recipe quality.
The blind comparison interface made the subjective part of the evaluation visible. It prevented a single aggregate score from hiding the difference between “well formed” and “actually good.” The frozen results and implementation are published with the product so the conclusion remains tied to the evidence.
What changed
The product now presents structural reliability and answer quality as separate outcomes. The results page reports the prompt-context reduction and schema gains alongside latency and review limitations. The Tasting Room invites direct comparison instead of asking visitors to accept an evaluator score.
That also changed the technical framing. Fine-tuning was no longer treated as a general upgrade to the model. It was a way to move a recurring response contract out of the prompt, with specific benefits and unresolved trade-offs.
What I learned
In this system, under these conditions, fine-tuning improved first-attempt structure and almost eliminated recurring prompt context. It did not establish that the answers had better judgment.
This changed how I evaluate fine-tuning work: measure the behaviour the training is expected to change, and keep structural compliance separate from subjective quality.