Optimizely Built Its Own Models. Here's Why That's Now Within Reach.

Matt Damon

09 Nov 2026 • 9 min read

At Opticon in September, Kevin Li talked about Optimizely building its own models, and my first reaction was that they couldn't have. Not because Optimizely lacks smart people. Rather, building a large language model from scratch is a game for companies that spend hundreds of millions of dollars on hardware, and Optimizely isn't one of them.

Then I read what they actually announced, and it's more interesting than "we built an LLM." It's evidence that a different kind of AI work has quietly become affordable for companies like Optimizely, and probably for companies a lot smaller.

What Optimizely actually announced

On September 1, Optimizely announced a family of purpose-built, post-trained models for marketing work. Partners have been calling them Mark-1, though Optimizely's own release doesn't use the name. The claim is frontier-level quality on marketing tasks at about 10x better cost efficiency than the leading general models, based on early testing. The models are listed as coming soon, and Optimizely hasn't said which base model they started from or how they trained it.

Alongside the models, Optimizely introduced Mark-Bench, a marketing benchmark it says it will open-source, and Mark-IQ, the data layer that feeds each organization's context to the platform. Imran Yousuf, Optimizely's Head of AI Engineering, summed up the strategy: "We believe post-training models is the long-term solution to the challenging cost vs. quality debate."

The key word there is post-training, and it's worth unpacking, because it's the difference between a nine-figure science project and something a mid-size software company can budget for.

Pretraining vs. post-training

Pretraining is where a model learns language and general knowledge from trillions of tokens of text. It's the expensive part. DeepSeek's widely quoted figure for its V3 model was about $5.6M, but that covered only the final training run. Analysts estimate the hardware behind it at well over $500M. Epoch AI has tracked frontier training costs growing about 2.4x a year, with the largest runs on track to pass $1B.

Post-training starts from a model someone else already pretrained and teaches it to behave a certain way. You show it examples of the work you want (supervised fine-tuning), or you let it attempt tasks and reward the good attempts (reinforcement learning). You're not teaching it language. You're teaching it your job.

That's what makes the economics work. When you post-train an open-weight model, you inherit hundreds of millions of dollars of pretraining for free and pay only for the last mile.

The training is the cheap part now

The numbers here surprised me. At current list prices from Together AI, supervised fine-tuning costs about $2 per million training tokens for a 70B-parameter open model. A realistic domain dataset of 50,000 examples, trained a few passes, comes to a few hundred dollars of compute. Smaller models cost less than that.

Reinforcement learning costs more because the model generates and scores attempts as it trains. For a sense of the upper bound, DeepSeek reported about $294K for the reinforcement learning stage of its R1 reasoning model, and Epoch AI's independent estimate is closer to $1M. That was on a 671B-parameter model. A company tuning a 30B model for one domain should land well under that.

So where does the money go? People and data. Here's my own back-of-envelope for a lean first year of a domain post-training program, built from public pricing, not from anything Optimizely has disclosed:

Line item	Rough first-year cost
A small team of ML and data engineers	about $1M
Training data and expert labeling	$250K to $600K
Training compute, including experiments	$100K to $500K
Building evals	$100K to $300K
Serving the models in production	$150K to $500K

That lands somewhere around $2M to $3M. It can go higher if you hire a large team, buy expert-labeled data at scale, or run heavy reinforcement learning on very large models. But for a focused program, under $5M is realistic. That's the scale of a serious product investment, not a frontier lab's budget.

Why this is possible in 2026

Two things changed. The first is that the open-weight models are good. Epoch AI puts the best open-weight models about four months behind the closed frontier as of this year, and many of the models in the size range a mid-size company would actually tune ship under permissive licenses like Apache 2.0 or MIT. Check the specific license before you build on one, because they vary more than they used to.

The second is evidence that a small model tuned for a narrow job can beat a much larger general model at that job. A June 2026 study tuned 3B-parameter models to extract structured relationships from text, and they outscored GPT-5.4 and Claude Sonnet 4.6. There's an important caveat: the frontier models in that study weren't tuned or given examples, and the small models wouldn't hold up outside their task. That pattern repeats across the research. Tuned small models win on narrow, well-defined work and lose on open-ended reasoning.

Marketing has plenty of both kinds of work, which is probably why Optimizely says the platform will route each task to the right model rather than asking customers to choose.

The hard part is deciding what "good" looks like

If the compute is cheap, what's scarce? The answer that keeps coming up, from practitioners and from the labs themselves, is evals: a reliable way to score whether the model's output is any good.

Hamel Husain, who has done more practical writing on this than anyone, put it bluntly: unsuccessful AI products "almost always share a common root cause: a failure to create robust evaluation systems." Anthropic's engineering team describes teams without evals as "flying blind."

For post-training, this goes beyond quality control. When you train with reinforcement learning, the scoring system is the training signal. Whatever your rubric rewards is what the model learns to do. In domains like math or code, you can check answers automatically. Marketing has no unit test for good copy, so somebody has to turn expert judgment into criteria a machine can apply.

That's why I think Mark-Bench may be the more important Opticon announcement. Optimizely's own write-up is candid about how hard it was. The first version was too easy: models passed more than 95% of roughly 7,000 criteria, so the team rewrote it with stricter ones. That's the real work, and it's the part nobody can buy off a price sheet.

Harvey, the legal AI company, is the clearest outside example of this approach working. In August it released a model post-trained on an open-weight base using about 1,750 legal tasks, each scored by an expert-written rubric. It nearly doubled the base model's task completion on held-out work. The investment was in the rubrics, not the GPUs.

Medicine has an even sharper example of how much the scorecard matters. OpenEvidence, the clinical answer engine, has published its case for training its own models and launched a family of them in September, though it hasn't said what they're built on. Its quality debate this year played out almost entirely in evals. In June, a Nature Medicine study from NYU Langone found that general frontier models beat OpenEvidence on exam-style questions, a clinician-written benchmark, and real physician queries. Weeks later, a preprint built on OpenEvidence's own user queries and graded by specialty-matched physicians found the opposite. The two studies used different questions and different graders, and they reached opposite conclusions. When a domain model's value depends on the eval, whoever writes the eval has a lot of say in the outcome.

Read the benchmark numbers carefully

I'm optimistic about where Optimizely is going, so it's worth being precise about what's been shown so far.

The headline result is that Optimizely's Agent Platform beat Claude Code 67% to 60% on Mark-Bench. That compares two whole systems, each a model plus the tooling around it. It isn't the new models against a frontier model. Another Optimizely recap from the same week reported 58% to 52%, and the task counts differ across Optimizely's own write-ups as the benchmark evolved.

The most interesting number is a different one. In Optimizely's Mark-Bench write-up, the same Claude Opus model cost about 88% less per task when it ran inside Optimizely's platform than when it ran on its own, and scored slightly better. That suggests a big share of the efficiency comes from the platform and context layer before any custom model enters the picture. That isn't a criticism. It's a sign the architecture is sound, and the post-trained models should compound on top of it.

The fair conclusion is that these are vendor-graded results on a vendor-built benchmark. The best thing partners can do is run Mark-Bench ourselves once it's published.

The marketing corpus is already out there

One reason this works for marketing in particular is that nobody owns the raw material. Good marketing is published by definition. Meta, Google, LinkedIn, and TikTok all run public ad libraries, and EU rules require the largest platforms to keep searchable ad repositories. Brand sites, landing pages, and product pages make up a large share of the web crawls that open models were pretrained on. A single monthly Common Crawl archive from last October held 2.6 billion pages from more than 38 million domains.

Optimizely's own benchmark makes the point. Mark-Bench was built largely from public material, including an open-source marketing skills repository, Zapier's automation benchmark, and Scott Brinker's State of Martech survey, plus Optimizely's internal agent library.

There's an honest limit here. The web shows you what shipped, not what won. Losing A/B variants, conversion rates, and brand guidelines rarely make it into public view. So the base models already know what marketing sounds like. What makes a model better at marketing is a well-defined standard for what good looks like, which brings us back to evals.

What this opens up for partners

For agencies and partners, this changes the conversation from "which model should we use?" to "what does good look like for this client, and can we measure it?" A few possibilities stand out to me. None of these are commitments, just directions I think are now realistic.

Evals as a deliverable. Building a solid eval set for a client's AI use case pays off whether or not anyone ever tunes a model. It's how you safely switch to the next model release, and it's the prerequisite for everything else here.

Small models beside big ones. In conversational products, the strongest case for a small tuned model isn't replacing the main model. It's handling narrow, high-volume jobs next to it: classifying intent, routing a conversation, enforcing guardrails, pulling lead details into structured fields. Those are exactly the narrow tasks where the research says small models shine, and they're where latency and cost matter most.

Building on open weights. OpenAI is winding down self-serve fine-tuning, closed to new customers since May and to existing ones in January, and it's shutting down its Evals platform at the end of November. If you're investing in tuning, the durable assets are your data, your evals, and adapters on open-weight models, not one vendor's pipeline.

The ops cost is real. Hosting models, versioning adapters, and retraining when a better base model ships all take people. But it's a cost a well-run agency can plan for, which wasn't true two years ago.

What we're doing at WOW

We're building Loominate, our conversational AI product, and every conversation it handles adds to a corpus of real questions, real intents, and real outcomes, used with our clients' permission. We expect that data to be useful in several ways: better retrieval today, evals tomorrow, and eventually post-training or small models for the narrow jobs inside the product.

We're not training models yet. But a year ago I wouldn't have put "post-train our own model" on a roadmap for a company our size, and now the math says it's within reach. Optimizely showed what that looks like for a platform company. The part I find most exciting is that the expensive ingredient isn't compute anymore. It's knowing what good looks like, and that's something we work on with clients every day.