Last month, we gave a posit::conf(2026) keynote on building correct, transparent, and reproducible data agents. The talk brings together many themes from this newsletter, so this week we’re sharing an annotated walkthrough. You can also see the full slide deck here.

Correctness vs. convenience

(Simon)

This morning, your VP pulled up a coding agent and asked a reasonable question:

How is traffic trending for our site?

The agent looked around the workspace, found a table that seemed relevant, wrote a SQL query against it, and returned a polished chart. Daily visits, it said, were down 64% over the last 90 days.

The variation looks a little tight, and the bend is a little sharp. Is there any possible reason we might see that bend other than a genuine change in site traffic?

It turns out that a version of Chrome released 45 days earlier fired the page-view event differently, so it was not aggregated correctly in the database. As people updated Chrome, it appeared that site visits were trending down. This was not even the table the data science team used to track site traffic.

But you are not there! Your VP sees this chart and takes it to the rest of the leadership team. They decide to restructure the organization and spend millions of dollars to reverse the trend.

A month later, somebody asks, “Could you recreate that chart? Let’s see how things are trending now.” The agent cannot find it. Many coding agents delete conversation histories after 30 days, and they often do data analysis in temporary files so they do not clutter the project directory. The source code for the analysis is long gone.

The coding agent offers to recreate the chart. It looks around the workspace, finds a table, and runs some SQL. The number was going up the whole time.

This is a fictional story, but we’e seen situations like this internally at Posit and at many of the organizations we work with. It is becoming normal to vibe-analyze data: ask an agent a question, receive a plausible answer, and move on without understanding how it was produced.

The tension between correctness and convenience is not new

Posit CTO Joe Cheng summarized the current moment this way:

AI has broadened the group of people who can convincingly do data analysis, but many may have no idea what data scientists go through to produce a trustworthy analysis.

Joe said this a month before the keynote, but he could have said it seven years ago. In 2019, the beverage company Conviviality disclosed that a spreadsheet arithmetic error had contributed £5.2 million to a £14 million forecasting mistake. When that “innocent spreadsheeting mistake” came to light, the business sank.

Or Joe could have said it 20 years ago, when a sign error in closed-source protein-modeling software sent a research field down the wrong path. A lab published landmark study after landmark study while other labs struggled to reproduce its findings. Only after another lab found a protein structure that was nearly a mirror image of one that the first lab had published did the first lab audit its source code. They found a negative sign where it did not belong and retracted five papers.

There were threats to correctness in data analysis long before AI. Posit’s stance has never been that people must sacrifice convenience for correctness. While preparing the talk, I found this quote in a blog post old enough to refer to itself as a “weblog”:

JJ was saying that a tool could be both correct and convenient. (By convenient, we do not mean 7-Eleven. We mean straightforward and intuitive.😜)

Can we take the same stance toward AI? Is it possible to make data analysis agents that are both correct and convenient for end users?

Why build AI tools at all?

(Sara)

This pull between correctness and convenience has been on our minds throughout the new, exciting age of LLMs. In 2025 and early 2026, AI started to feel really, really convenient for data analysis. That made us worry about correctness.

In August 2025, when we released Databot, our state-of-the-art exploratory data analysis agent, we did so under what was essentially a warning label. The release post was one of the first things I worked on after joining the AI team, and it gave me the chance to do something I think I am particularly good at: be pessimistic.

We were all really excited about Databot. It felt like flying through your data, gathering insights faster than you thought would ever be possible. It was precisely that convenience that worried us. People might trust it because they were having so much fun, even when it was wrong. So we wrote an entire article detailing basically every way we thought Databot might fail.

If you were at posit::conf(2025), you might remember Joe Cheng describing it as both the most exciting and the most dangerous software he had worked on in his 30-year career.

That raises a fair question, one we contemplate ourselves on a weekly basis: If AI causes all this chaos and makes all these mistakes, why are we doing this? Why involve ourselves at all?

There are many answers. We’ll focus on one.

Curiosity and exploration

Before joining the AI team, I taught data science. One thing that always struck me was what brought people with very different backgrounds into the field. The thing that united many of them was a sense of curiosity. Maybe they were curious about a scientific field, computers, statistics, or how R and Python themselves work. Data rewards that curiosity: There are datasets on almost everything.

But even as an undergraduate who knew some statistics and a little R, I often felt like I was looking at an ocean from above. The dataset was right in front of me, and I knew there was so much going on, but I just couldn’t see it.

As a graduate student, I learned the tidyverse and more about data science. As I got better at using those tools, I finally felt like I had what I needed to ask and answer questions about my data.

The right tools let us see below the surface. They enable that curiosity.

That same curiosity motivates our work now. We are curious about how these models work, how we can build things around them to make them work better, and, maybe most importantly, how we can turn them into tools that open new worlds of analysis. For us, AI is a continuation of work we have always done: finding new ways to explore data.

The models got better

I also wanted to spend some time on what has changed since August 2025, when we released that warning-label image. One major thing happened: The models got better.

About a year before the keynote, we ran an experiment to understand how well models interpret plots. We showed them plots based on a transformed diamonds dataset and asked them to describe what they saw.

The models gave answers like, “There is a strong positive relationship between carat and price.” That sounds plausible until you look at the plot, which shows the opposite. Behind the scenes, we had manipulated the data to reverse a canonical relationship, so larger diamonds were, unintuitively, less expensive.

We wanted to know whether the models could see what was in front of them and interpret plots that contradicted their expectations. A year ago, the answer was largely no.

In November 2025, even the best models of the time were abysmal at this task. They said what they expected to see instead of paying attention to what was actually happening in the plot.

We tried all sorts of interventions to improve the bluffbench scores, and absolutely nothing worked. Then we waited six months.

By May 2026, the major AI companies were releasing models that were noticeably better at the task. By September, the numbers had jumped dramatically: The models could interpret these counterintuitive plots.

The models got better. That does not mean data science is solved.

Models still make mistakes

After that experiment, we asked another question: Can models identify data-quality issues in visualizations? In bluffbench2, we again showed them plots and asked what they saw.

If you look carefully, there’s a suspiciously straight line through the cloud of points. If we really cared about this data, that’s something to investigate: Did something weird happen when the data was created? We wanted to know whether models would do the same.

It turns out they don’t really. Even the leading models were not very good at this task. They struggle to notice subtle data-quality issues.

But there’s another problem. Data analysis is not just you, plus a good model, plus your data. It also depends on context that is not necessarily in your CSV file, warehouse table, or JSON file.

All data is constructed. It all has context. That context might live in a data dictionary, a Markdown file, or a semantic view. Maybe it just lives in the head of a coworker who has been at the organization for 15 years, knows absolutely everything, and has never written a single thing down.

That context is not inherently inaccessible to models. We could give it to them. But it has to be in the right format, it has to be curated, and it has to itself be correct.

So the models have gotten better, but data analysis with AI is not solved. The question then becomes: How do we make AI-involved analysis correct?

How do we make AI for data analysis trustworthy?

(Sara)

Here’s a common thought about AI: It’s untrustworthy, but useful. To make it trustworthy, we need to add in a person.

Here’s a simple version of what that might look like. The agent or model does stuff, probably writes a bunch of code. A person then looks at that code and verifies, approves, or redirects as needed. That feeds back into the agent. This intuitively can feel really appealing. The agent gets to do what it’s good at, the person does what they’re good at, and there’s an expert there to keep everything from going off the rails.

And so it’s kind of tempting to apply this pattern everywhere. Any time the agent might make a mistake, we have a person there to clean it up and make sure nothing bad happens. We call this relatively simplistic view the slap a human on it approach.

In this approach, you don’t really think very much about what the person will actually do. How will they apply their expertise? What is the purpose of their involvement? How will we keep them from technically being in the loop but without actually applying their expertise?

I’m being a bit facetious, but this isn’t that far from how we wrote risk mitigation with Databot a year earlier. We said that Databot and LLMs were not at a point where users could abdicate responsibility. You still needed all of your data skills to catch errors, interpret results in context, and avoid being misled by confident but incorrect claims.

And all of that is still true! But we never said how this was supposed to happen. We assumed that if an expert was there, they would catch the mistakes.

But just adding a person doesn’t really guarantee a better result. Human–AI combinations do not reliably outperform the better of the human or AI working alone.

People also get tired of approving requests, and then they stop reading them. Think about the last time your coding agent showed you a command to approve. Did you carefully read it, or did you hit approve as quickly as possible and feel annoyed that it even bothered to ask?

Human decision-making is not fixed. How much you trust the system, how much you trust the people who built it, the timing of the information, your cognitive load, and how much your coworkers trust the agent can all influence the decision you make. Even if you are an expert who makes the right decision outside an interaction with an agent, adding the agent might change that decision.

So how do we design the entire system to produce correct work? How do we build trust and correctness into the agent itself, and design interactions with users that actually support their expertise?

That maps onto two principles we returned to throughout the talk:

1. Help the agent be correct.
2. Make it less bad when the agent is wrong.

Posit Assistant

(Simon)

Posit Assistant is Posit’s coding and data science agent. It is available in RStudio, Positron, and the terminal. When it is running inside an IDE, it works in the same R or Python session you’re using. You can use whatever model you want from whatever provider you have access to.

For the demo, I returned to one of my first experiences of seeing below the surface with data science. In an introductory statistics class, I analyzed flight data to understand whether I could book tickets that were less likely to be delayed. I used ggplot2 and dplyr, pushing data frames around and visualizing them.

I asked Posit Assistant to find an R package with Houston flight data and retrieve data from the previous year. It found anyflights, read its documentation, and generated the correct call.

Next, I asked it to grab data from the same time the previous year. The agent read the package documentation and so was able to write the code with the right arguments the first time.

We have seen that many coding agents have a superficial relationship with data quality, and we wondered whether we could do better. Posit Assistant proactively recommends cleaning the data and has a dedicated data-cleaning mode. It runs scratch code to look for missing values, outliers, and similar issues. Once it finds them, it surfaces questions to the user in an interactive dialog.

In the demo, the agent found missing values in a relevant column, then found some flights with departure delays of multiple days. I chose to keep both: Those outliers represented what really happened.

At the end of cleaning mode, the agent wrote a persistent file in the project directory to ingest and clean the data. When someone revisits the analysis, they can reproduce those steps.

Posit Assistant also shows plots inside the conversation. It is nice to be able to see them, but we also think it is necessary: Our evaluations show that agents cannot reliably interpret plots. When one carrier appeared unusually delayed, it checked the sample size and found only 32 flights, suggesting that we should not read too much into the pattern.

Once the analysis was complete, I asked the agent to turn it into a persistent Quarto report.

I have one more “little zoomy zoom”: The entire conversation cost five cents.

Part of our mission at Posit is to provide tools to people regardless of their economic means.

Posit Assistant is designed to be token-efficient and can use capable, inexpensive open-weight models such as GLM 5.3 Flash through Posit AI Pass. We spend a lot of time watching the open-weights model space for models on the Pareto frontier: capable of data analysis and orders of magnitude cheaper than frontier models.

Returning to the two principles, part of helping the agent be correct is, quite literally, asking nicely. We say, “Pretty please pay attention to data cleanliness. Pretty please be open to uncertainty when carrying out data analysis.”

We also design interactions around the user’s expertise: Cleaning mode asks about the data-generating process, plots remain visible, and the user shares the agent’s R or Python session.

Posit Assistant assumes that the user is a data scientist, analyst, or someone else who writes code and wants to work in an IDE, close to that code. But what if the user is not a data scientist? How do we help them get correct answers, too?

Introducing commons

(Sara)

Let’s return to our vibe-analyzing VP. I think it’s easy to say, “They just shouldn’t be doing this kind of analysis.” But let’s be on their side for a moment. They had a question, they wanted it answered, and so they’re going to reach for a tool that gives them an answer.

The problem is that they reached for the wrong tool and it gave them the wrong answer.

The data team already has vetted, maintained code that computes site traffic correctly. We call this trusted code. Trusted code might live in a package, Shiny app, Quarto document, report, or dashboard.

Instead of having the model write bespoke code from scratch every time someone asks a question, what if it could use that trusted code?

This is one of the core ideas behind a new open-source package that we’re excited to share with you all today. It’s called commons.

commons helps data scientists build trustworthy data agents for their collaborators. It is an open-source R and Python package built on the ellmer, chatlas, and shinychat stack.

(Simon)

Here is how the VP’s analysis works with a commons agent. When the VP asks, “How is traffic trending for our site?” the first thing the agent does is search for a trusted calculation. If it finds one that answers the question, it invokes that calculation directly.

The calculation returns a plot and a green shield indicating that the answer came from a trusted calculation. commons adds that marker deterministically; the agent itself does not control it.

What makes coding agents so convenient is that they are everything tools. They will write their own R, Python, or SQL code to answer a question. Trusted calculations cannot cover every question, so commons has a fallback path.

When no trusted calculation exists, the agent can use context authored by the data team. commons verifies any citation against that context before displaying its blue citation marker.

If there is neither a trusted calculation nor supporting context to cite, the agent can write code, but commons displays a yellow warning so the user knows to treat the answer cautiously.

Here’s a diagram of the various ways the agent can come up with answers and how the trust relationships are mapped.

So far, we’ve been talking about the VP’s experience with the commons agent. What is it like to work with commons as a data scientist? There are three main loops: build, deploy, and improve.

With data collection enabled, data scientists can review cases where the agent had to write its own code. If those cases share a pattern, they can add trusted code that moves future responses up the trust ladder, making the agent more correct over time.

Returning to our two principles, commons helps the agent be correct by letting it directly use trusted code. It makes mistakes less bad by telling the user when the agent did not directly use trusted code, and by moving responses up the trust ladder over time.

Other use-cases for commons

(Sara)

The VP example is business intelligence, but commons is not only for BI or vice presidents. It’s for any data person who wants to build an agent that gives correct answers to their collaborators.

We’ve made a sample clinical trials agent to highlight this functionality. Its data is simulated. The trusted code is a TLG catalog: code written and trusted by data people to work on this type of data and produce correct answers. You can ask it questions such as, “Plot Kaplan–Meier curves.”

It is still very bad to be wrong

(Sara)

It’s now so easy to ask any kind of agent a question about your data and get a wrong answer that looks completely plausible.

The problem is not just that the answer is wrong, and it’s also not just that you might be unable to reproduce it or have no idea how it arrived at the answer.

These are bad enough, but there’s another problem. If we become accustomed to, say, 15% of our numbers being wrong, we might lose trust in analysis itself and its ability to tell us things about the world, or become used to numbers just being a bit wrong all the time.

But correctness matters, especially in particular industries, and Posit has always built tools to support correct analysis. We do not think that standard should change just because AI is here.

But AI has expanded who can get answers from data. Now anyone can ask a question about their data. That changes how we need to think about the infrastructure for building correct, transparent, and reproducible agents.

With commons, a data team establishes which code and calculations it trusts and provides the organizational context the agent needs. The team can give that agent to someone who might otherwise perform a vibe analysis. That person keeps the convenience of asking questions in natural language, while the answers are grounded in the data team’s trusted work and can be traced and reproduced.

That is where data scientists and analysts come in. This infrastructure must be built by people who understand the data, calculations, and organization. commons helps them build it and make it available to others.

Looking forward

Looking forward, we have been experimenting with something called Canvas.

Canvas is a next-generation agent experience. It assumes that you might not always want to look at the code. That does not mean the code is not there when you need it, but you might not always want to work in an IDE.

This frees up real estate for expansive, new agent-collaboration interfaces. Canvas feels a little aggressively agentic to us, in the way Databot felt a year ago.

Recent past newsletters

• New releases from ellmer, shinychat, and commons
• You probably don’t want to fine-tune

Subscribe via RSS

Last week, three packages in Posit’s open-source AI stack shipped significant releases. We introduced commons 0.1.0, and ellmer and shinychat both received substantial updates. These releases are part of a broader effort to make it as easy as possible to build modern chat applications in R and Python.

In this newsletter, we’ll take a quick tour of what’s new.

Introducing commons

commons, a new framework for building trustworthy self-service data analysis agents in R and Python, is now on CRAN. Read the full announcement here.

The Python package is currently in a pre-release beta stage, with more features arriving over the next few weeks.

If you’re a data analyst, data scientist, statistical programmer, or other data practitioner, you likely have extensive domain knowledge and a collection of trusted code that you already use in analyses, apps, reports, and packages. The core idea behind commons is that we can leverage this trusted code to improve an agent’s correctness.

A commons agent first searches for a trusted calculation. If it finds one that can answer the user’s question, it can run that vetted code and the answer is deterministically marked as verified.

If it doesn’t find a trusted calculation, the agent searches trusted context before writing custom R, Python, or SQL. The answer is either given a citation or marked as “untrusted,” depending on whether the agent provides a verified citation that supports its approach.

The model doesn’t decide how trustworthy its answer is. commons assigns each label deterministically based on the analysis path taken by the agent.

commons also ships with an agent skill to help you create a commons agent and functions for analyzing your users’ conversations.

shinychat v0.5.0 (R) and v0.7.1 (Python)

shinychat v0.5.0 for R and v0.7.1 for Python bring together more of what you need to build a complete chat application. Several of these shinychat updates also made commons possible!

Read the full blog post here. There are many more updates worth checking out.

page_chat()

Use page_chat() instead of the bslib page_*() functions when you want the chat to be the center of your application. page_chat() creates a full-window, chatbot-oriented layout with support for navigation pages, conversation history, an artifact drawer, and more.

Conversation history

Conversation history is enabled by default when you use chat_server() in R or Chat(client=...) in Python, allowing users to start a new conversation, switch between saved conversations, search them, rename them, and delete them.

Readable tool calls and citations

This shinychat release also includes several improvements for understanding how a model arrived at its response.

One such improvement is readable tool calls. By default, related tool calls are grouped into compact, single-line “activity rows”, keeping them from overwhelming the conversation. You can still inspect the individual tool calls by expanding a row.

shinychat also displays citations returned by providers’ built-in web-search and web-fetch tools.

ellmer 0.5.0

ellmer 0.5.0 is now on CRAN. ellmer makes it easy to work with LLMs from R.

Read the full announcement here. Many of the features made available in this release are also available in recent releases of chatlas, ellmer’s sibling package in Python.

Citations

When a model uses a supported built-in web tool, ellmer now returns and displays the provider-supplied citations. This works with claude_tool_web_search(), claude_tool_web_fetch(), google_tool_web_search(), and openai_tool_web_search(), helping you identify the sources behind the model’s answer.

chat <- chat_openai()
chat$register_tool(openai_tool_web_search())
chat$chat(
  "What is the most recent version of ellmer on CRAN? Look it up."
)
#> The most recent CRAN release of **ellmer** is **version 0.5.0**, published
#> **September 4, 2026**.
#> ([cran.r-project.org](https://cran.r-project.org/package%3Dellmer))[1]
#>
#> Sources
#> [1] CRAN: Package ellmer: https://cran.r-project.org/package%3Dellmer

Tokens and costs

Managing tokens and costs is an important part of working with LLMs.

Ever want to know how many tokens an input will take before sending it? For supported providers, you can now use chat$token_count() to estimate input token use. Instead of actually sending the request to the model, it sends the request to the provider’s token-counting API.

chat <- chat_openai(model = "gpt-5.6-luna")
prompt <- content_pdf_file("example-document.pdf")
chat$token_count(prompt)

#> [1] 252

This estimates only the tokens used by the input and does not predict the number of output tokens, so it won’t represent the total round-trip count.

Companies frequently release new models and change their prices. Use the new function models_update_prices() to download and cache the latest pricing data from the ellmer GitHub repo. Cost estimates reported by token_usage(), Chat$get_cost(), Chat$get_tokens(), and printed Chat objects use this data.

Send files to the model

It’s often useful to send files as part of a chat. You can now send CSV, Markdown, code, and other text-based files to a model with content_document_file() and content_document_url(). For large files or files reused across multiple turns, if you’re using chat_openai(), chat_anthropic(), or chat_google_gemini(), use chat$file_upload() instead. It uploads the file once and returns a reference for $chat(). This avoids repeatedly sending the file and reduces token usage and cost.

Solid improvements for custom agents

Taken together, these releases make it easier to build more complete custom agents. ellmer manages model interactions in R, shinychat provides the user-facing chat interface, and commons adds a framework for data analysis agents that builds upon these two.

Recently, there’s been a lot of talk about fine-tuning models. The reasoning usually goes something like this:

• There’s some task that needs to be done regularly.
• Today’s frontier models can do it quite reliably, but it’s expensive and the bills are starting to rack up.
• Cheaper models can’t quite do the task reliably, but a fine-tuned variant of a small, open-weights model might be able to just as cheaply.
• Therefore, you should fine-tune models for tasks that your organization does.

I’ve seen a number of blog posts cited in support of this idea.

It makes sense that fine-tuning is appealing! AI is getting expensive, many players in the space cannot make the privacy guarantees we’d hope, and just as you start to rely on one proprietary model, it’s phased out in favor of a newer release. That said, my reaction is that this approach seems more engineering-intensive and, likely, more expensive than alternative approaches. I’ll try to make that case here.

What is fine-tuning?

Before we go further, it’s probably worth quickly outlining what I mean by fine-tuning. In short, fine-tuning means training an existing model further to improve its performance on a particular task.

Large language models contain huge arrays of parameters. Input, usually in the form of text, is turned into arrays of numbers, and those arrays get multiplied a bunch of times to form the output, which is then typically converted back into text. Those multiplications are quite computationally intensive, and the array of parameters is itself quite large. For example, the smallest models that are beginning to be able to reliably complete basic agentic work contain 8 billion parameters (in short, “8B”).

Fine-tuning changes a model’s behavior, either by updating the existing parameters, or by training a smaller set of additional parameters that work alongside the originals. The goal of fine-tuning is, broadly, to increase the performance of a model on a given task while minimizing the impact on the model’s ability to do other tasks.

Fine-tuning is hard

It is very, very difficult to successfully fine-tune a model. I don’t claim that it’s impossible, but it is a substantial engineering effort, requiring the careful attention of dedicated scientists with access to today’s frontier models across several weeks, just for the first edition of the model. To show why, let’s revisit each step of that process.

What model should you start with? In short, you want the smallest possible model that has the ability to learn to do the task reliably while retaining sufficient general intelligence. It is very hard to predict when or how capabilities will emerge during fine-tuning without first fine-tuning, meaning that the choice will need to be revisited several times once you’ve made a first go at the remaining steps.

What data gets used for training? Once you’ve chosen a model, you need to find some training data to fine-tune it with. What should you use? Ideally, you would use the real data that represents the task and its inputs. However, in order to train on that real data, you typically need the user’s consent, and it’s likely that you won’t have that consent.

And then, when you do have user consent and choose to train on that data, you now have a mandate not to overfit. This is because if the model you’re fine-tuning internalizes the data you’re training on and is able to (even hazily) recollect it, you’ve now exposed that data to any users of the model.

The other approach, then, is synthetic data, created by asking a model to generate a bunch of scenarios that resemble the real inputs and then demonstrate how to carry out the task in those scenarios. But this type of synthetic data has its own problems: notably, the models creating the data have their own set of tics that make the data unrepresentative of real-world data. The Elias in the Lighthouse effect is a notable example of this phenomenon. A wide variety of models will, when asked to write a story, write about a lighthouse keeper named Elias Thorne. On its own, the story of Elias Thorne might be informative training data. However, 1,000 stories about a lighthouse keeper Elias Thorne are not.

How do you nudge the weights? Let’s assume you do have a diverse, representative set of training data to work with. You’ll now need to decide how, mechanistically, to change the behavior of the model—do you need a full fine-tune, or will a LoRA be sufficient?

Next, you’ll need to tune a set of 5-10 hyperparameters. Notably, because they are hyperparameters, there is no generally good “magic number” for them, and instead you’ll need to try a bunch of values and see what works. You’d make some guesses, see what happens, then generate some hypotheses on what a given change to those parameters might do, then try out a different number and see if it has the desired effect. Every time you’re figuring out what to try next, you’ll be reading a ton of test cases and trying to observe general failure modes exhibited in this intermediate draft of your fine-tuned model. That might change the initial choice of model you’re fine-tuning, and it might change which subsets of the training data you’re exposing to the training process.

How do you measure success? You need a reliable way to tell whether each version of your model is actually improving. That is especially tricky for free-text tasks, because two answers can be equally good even when they differ syntactically. Because of this, you’ll probably want an LLM-as-a-judge system, where another model compares the responses to a target and grades according to a rubric you’ve supplied. LLM-as-a-judge systems are themselves quite hard to design correctly, and you’ll be reading a bunch of that judge’s grading transcripts, too.

It’s also worth mentioning that getting your fine-tuned model to score well on your own benchmark is not the hard part. Many of these fine-tuning writeups include a plot along these lines:

When I see that a single-digit-billion-parameter fine-tuned model scores better than Fable 5.1 or GPT 5.6 Sol on an evaluation, I interpret that as evidence that the evaluation is not meaningful. In my own experience, it is not (comparatively) hard to get a small model to score very well on any given benchmark. What’s much more difficult is preserving the broad intelligence of a model while driving the evaluation score up. A meaningful evaluation can do both at once, measuring task performance under a realistic, broad distribution of possible task configurations. It is very hard to author meaningful evaluations, especially for models as capable as those that exist today.

But let’s say you’ve made it this far! You found a good model to start from, a diverse set of training data that you’ve obtained consent to train on, and a meaningful way to measure progress. Now, it’s time to put it in production.

Fine-tuning is expensive

Fine-tuning can seem appealing because the actual fine-tuning part of the process can be relatively inexpensive, likely on the order of a few dollars to a few hundred dollars.¹

However, actually deploying the model to do the task you trained it to do is likely to be substantially more expensive.

…because hosting is expensive

Hosting a model yourself is more expensive than using a similarly capable model hosted by someone else unless you have extraordinary volume. Frontier-model providers like Anthropic and OpenAI serve extraordinarily large volumes, keeping their compute almost always near max capacity. It’s likely that the same won’t be true for your fine-tuned, self-hosted model.

As an example, let’s say I host Gemma 4 26B A4B on a single H100. I can rent a high-availability H100 GPU instance at $0.0833 a minute, or $44,000 a year. Assuming a coding-agent-like workload,² that would buy about 96 billion Sonnet 5 tokens, and Sonnet 5 is a much more capable model. To break even, this fine-tuned model would need to serve about 182,000 tokens per minute, 24/7, throughout the year.

It might be possible to serve that many tokens. The hard part is finding enough demand for one task to keep the model busy, and that demand needs to be relatively smooth. If the model sits idle, you’re still paying for it. If demand spikes, you either rent a second GPU or accept worse performance for users.

…and you must host

The other suggestion here is that, if the model is small enough, users who need to do the task can just download the weights and run it on their laptop or a dedicated workstation.

The effectiveness of this argument depends on the kind of task to be done. However, it’s worth considering the potential for this to be a very unpleasant user experience compared to just using a model that someone else hosts. For example, let’s say my colleague does some task for an hour every week and there’s some model small enough to run on my laptop that’s capable of doing the task. In order to use the model for that task, my colleague would need to download the 4GB or 8GB or 100GB or whatever of weights. That user would also need to be capable of configuring the serving of the model, and they’d need to make that happen every time they started the task. If that model was accessed through a tool that the user was using regularly anyway—Claude Code pointed at a local ollama model, for instance—they’d need to remember to switch the model over in the application’s settings. If that model was accessed through a different interface, the user would need to use a different interface than they normally use LLMs with for that task specifically. Neither of these are good UX.

Even if the software for locally serving models got much more pleasant than it currently is, it’s hard to compete with the experience of pay-as-you-go for the Everything Tool that wraps the Everything Model.

Your fine-tune will quickly fall behind

Let’s say you successfully fine-tune a model based on some fictional model Kuen 3. Then, the following week, Kuen 3.5 is released. 3 of the 100 most capable LLM scientists on this planet worked on it. It’s almost as good as your fine-tune on that specific task, and it’s also broadly capable at a very broad array of tasks.

Fine-tuning is not a boat that is lifted by the rising tide of broader AI progress—in order to take advantage of the Kuen 3.5 release with your fine-tune, you’d need to restart that process of choosing parameters and observing fine-tuning runs. The underlying architecture of the model may have changed, and thus the approaches that you used to fine-tune Kuen 3 might not work for Kuen 3.5 on your first try.

What you should do instead

Instead, the boat that is lifted by a rising tide in this context is plain old prompt engineering. (POPE, as Joe Cheng calls it.) Choose a model that’s available to you, that fits your price point, and seems broadly capable across a wide variety of public benchmarks. (Extra points if the vibes on the model from people you trust are good, and extra points if someone else is serving it.)

Then, put together a short prompt telling the model how to do the task, perhaps inside of some coding agent harness like Claude Code (or, hey, Posit Assistant), and see what it does. Adjust the prompt to tell it how to do things correctly that it tends to trip up on, and iterate from there. If you’ve already put together an evaluation as part of your fine-tuning process, you could even reuse that! My colleague Sara and I have written about prompt engineering in the past if you’re interested in learning more, once about the more specific task of POPE for teaching LLMs about R packages but in relatively generalizable ways, and once focused on deciding between the popular ways to deliver context to coding agents.

Recent past newsletters

• How to choose a model
• Keep track of your data exploration with Posit Assistant’s EDA log

Subscribe via RSS

How do you know which model to use and when? Often, it’s not a question of which model is best, but of which model suits your task and needs at a given time. You might switch between models for different projects (package development vs. data analysis), or even within a single project (planning vs. implementation). Different tasks require a different mix of cost, token usage, speed, intelligence, and capabilities.

If you want our most durable, high-level advice: start with the most expensive model you have access to from either OpenAI or Anthropic.¹ Then, once you have a sense of the “ceiling,” try less expensive models and see how they compare. Developing a feel for what’s possible with LLMs will help you make better decisions about trade-offs.

That said, we’ll try to tackle this question more thoroughly in this post.

The model landscape

Currently, Anthropic and OpenAI set the bar for AI capabilities. Google is often discussed as a third frontier lab, though its current model lineup is less competitive.

Anthropic and OpenAI each release a “family” of models: Claude and GPT, respectively. The most capable models in both families are also the slowest and most expensive. Conversely, the least capable models are the cheapest and quickest. Other labs tend to follow this same pattern, releasing a set of models with different trade-offs along the cost-performance curve.

Models within a given family tend to share a similar shape of intelligence, with related capabilities (relative to model size), shortcomings, and idiosyncrasies. For example, Claude Fable 5, Claude Opus 5, and Claude Sonnet 5 often use the same turns of phrase and are quite good at writing code and debugging it. Models from different families can have different shapes of intelligence even when their benchmark scores and prices are very similar. For example, GPT-5.6 Terra and Claude Sonnet 5 are priced similarly and comparably capable at agentic coding, but Terra doesn’t “see” data visualizations as well as Sonnet, while Sonnet doesn’t communicate as clearly as Terra.

Other labs release models that score nearly as high as models from Anthropic and OpenAI on benchmarks. However, these evaluation scores can be deceiving. Labs can now train models to optimize for benchmarks (“benchmaxxing”). These models score well on benchmark-shaped tasks, which tend to be highly autonomous and “tricky,” but can fail to generalize to real-world tasks, which often involve more ambiguous requests. Kimi K3 and GLM 5.2 offer a counterexample: they score well on benchmarks, and we’ve also found them exceptionally well-rounded and intuitive. We recently introduced both models to Posit AI.

In previous newsletters and blog posts, we’ve shared results from targeted evaluations. Here, instead, we offer an approximate and entirely vibes-based comparison of these model families’ characteristics.

For data science applications broadly, you can loosely think of the relevant score as the average of the agentic coding and vision scores we’ve assigned here. Beyond writing R and Python code, models need to be able to interpret plots accurately and faithfully.

Find the model that fits your task

Much of model choice is constrained by the models you have access to. Your organization may only allow a certain provider, or you might not want to pay multiple (possibly expensive!) subscriptions just to have access to all the top models.

If you do have your pick, however, here are some quick guidelines, partially shaped by what models are currently available through Posit AI.

The best open-weights model, especially for data analysis: Kimi K3.

As Simon wrote in the Kimi K3 and GLM 5.2 in Posit AI announcement post, “Kimi K3 is currently the most capable open weights model out there. In our internal testing, it feels somewhere between Opus 5 and Fable 5, and is notably well-rounded compared to other open weights releases.”

It also ranks near the top in bluffbench2 (13.46%, compared with 16.35% for the tied top scorers, Gemini 3.5 Flash and Claude Fable 5), which evaluates models’ abilities to spot subtle data quality issues in visualizations.²

A highly autonomous model for a complex coding or data task when cost and speed aren’t a concern: Claude Opus 5, Claude Fable 5, or GPT-5.6 Sol.

These are the top-of-the-line models from Anthropic and OpenAI. They are expensive and relatively slow, but can be worth using for ambitious or highly autonomous work.

A strong open-weights model for coding when you don’t need vision: GLM 5.2 (from Z.ai).

“GLM 5.2 excels at agentic coding and less so at data analysis tasks.” It is much less expensive than the proprietary models it resembles for coding tasks, but it does not natively support image inputs.

A middle-tier generalist for coding or data analysis: Claude Sonnet 5 or GPT-5.6 Terra. Both are capable across coding and routine data analysis, support vision, and are less expensive than their respective labs’ higher-tier models.

Good plot or image interpretation: One of the Gemini 3.x Flash models (3.7 was released on August 13).

This model series is particularly strong at vision and has performed well on bluffbench (3.5 Flash) and bluffbench2 (3.5 and 3.6 Flash).

Fast answers from an Anthropic model for a task that is not particularly complex: Claude Haiku 4.5.

Fast answers from an open-weights model for a task that is not particularly complex: Gemma 4.

Assorted notes from August 2026

In late summer 2026, a few developments feel notable, but, as with much in the AI world, who knows how long these observations will hold.

• Google Gemini currently does not have any models that perform near the frontier. With the releases of Gemini 2.5 Pro (June 2025) and the Gemini 3 series (November 2025), Google seemed positioned as a third frontier lab. However, it’s been quite a while since they released a frontier model, and they’re now meaningfully behind. As of the time of writing, Google says Gemini 3.5 Pro is still testing with partners. Meanwhile, the company has begun training Gemini 4 and says it is excited by the progress.

• For a year or so, it seemed like Anthropic was solidly ahead of OpenAI in agentic coding. However, since the Claude 4.6 releases, it has become less clear that Anthropic is meaningfully ahead of OpenAI. For one, Anthropic’s current high-end models use newer tokenizers that, according to Anthropic, produce roughly 35% more tokens for the same text than their predecessors. This means the same listed price per token does not necessarily translate to the same cost for comparable text. Further, in our experience, the Claude series has become increasingly token-hungry and difficult to communicate with. At the same time, OpenAI’s models have a notably clear, concise communication style compared with the Claude 5 series. Anthropic is still likely ahead on autonomous, long-horizon coding, but OpenAI no longer feels behind for day-to-day software engineering and data science.³

• There are now a number of balanced, well-rounded open-weights models relatively close to the closed-weights frontier. Kimi K3 and GLM 5.2, in particular, combine strong capabilities with a more pleasant, intuitive feel than their predecessors. While earlier open-weights models were just as close to the frontier in benchmark scores, some newer releases are notably more well-rounded and respond to prompts about as effectively as proprietary models. These releases are also priced at a steep discount compared with the proprietary models they most resemble.

Recent past newsletters

• Keep track of your data exploration with Posit Assistant’s EDA log
• Which models are best at spotting subtle data quality problems?

Subscribe via RSS

The log summarizes findings for different areas of exploration and keeps track of next steps. To use the log, run the /eda-log slash command after you’ve started the EDA process.

Why we made this

Exploratory data analysis, the open-ended orientation to your data that often comes before anything else, can be a branching, nonlinear process. There are many questions you can ask of your data, and new areas of inquiry can open with each question you ask. Because of this, it is often hard to keep track of what you’ve looked into, where that code lives, and what you want to explore next.

Historically, the EDA process was limited by how quickly you could write code and interpret the output. Coding agents like Posit Assistant lift the first of those constraints. They can carry out EDA far faster than you can on your own, which can exacerbate the issue of keeping track of what you’ve explored.

This speed also introduces a new problem: the point of EDA is typically for you, the human, to understand your data, but coding agents can produce output faster than you can absorb it. If the agent completes an analysis but you haven’t understood the insights in the data, the exploration process hasn’t really happened.

Posit Assistant already has various features that tackle this problem, including an exploratory mode of interaction where it runs shorter turns and stops more frequently to involve the user.

The EDA log is another lightweight tool for the same goal. It keeps a running summary of what you and Posit Assistant have explored, helping your understanding keep pace with the agent’s and giving you a clearer picture of what’s already been done.

Details

Here’s what the EDA log looks like in action:

At a high level:

• When you run /eda-log, Posit Assistant will create a log for the exploration done in the conversation so far. The log then opens in the editor area in Positron.
• The underlying logs are stored as YAML files in .posit/assistant/eda-logs/, next to where plans are stored.
• Posit Assistant is instructed to loosely keep the log up to date as the conversation progresses, but you can also manually trigger an update at any time with the “Refresh” button.
• Clicking the arrow next to an area scrolls you back to the spot in the conversation where that insight originated, so you can revisit the code and context that produced it.
• Suggested next steps appear as clickable text. Clicking one sends it to Posit Assistant as your next message.
• The creation of an EDA log is always user-triggered. Posit Assistant will never create one on its own.
• The feature is currently only in Positron, but will come to RStudio soon.

Recent past newsletters

• Which models are best at spotting data quality problems?
• How to choose between AGENTS.md, skills, and MCP servers

Subscribe via RSS

Huh. It mostly looks normal, except there’s a few points perfectly aligned with what looks to be a “fitted” line. You dig into it a bit more, and realize that the rows from one study site have their cholesterol values imputed. You set them to NA and go along your way.

Would today’s frontier LLMs catch such an oddity? We designed an LLM evaluation to help us answer this question. As it turns out, LLMs mostly miss these sorts of artifacts:

During exploratory or open-ended data analysis, Posit assistant “only runs a few bits of code at a time, then summarizes what it found and suggests next steps”. This is motivated by our stance that, for now, a data scientist should mostly keep pace with and understand what the agent is doing when analyzing data. This stance was initially informed by our observation that last year’s frontier models tended to see what they expected to see when visualizing data. While LLMs have since become much better at interpreting counterintuitive plots, bluffbench2 shows they still lag behind human data scientists in interpreting data visualizations. As such, we are still cautious on the prospect of highly autonomous data agents.

How the eval works

The eval harness is a relatively generic coding agent harness, similar to that of Claude Code or Posit Assistant. The agent has a tool to run R code in a persistent REPL and some vague prompting about data analysis:

You are an AI assistant embedded in the user’s data science IDE. You can read and modify files in the user’s workspace and execute R code in their active session, including rendering plots. Prioritize correctness and clear communication…

In each sample, the agent first carries out a few “lull” turns, making a couple plots and tables unrelated to the eval. Short user messages like “load in the csv in this folder” are decorated with “System Reminders” and other noise like that injected by popular agent harnesses.

After a few turns, the agent is asked to produce a data visualization that includes a subtle visual artifact that could feasibly result from a real data-generating process. The artifacts span a range of realistic data quality issues: stuck sensors, bad joins, points imputed onto a line, swapped columns, pseudoreplication, differing units, etc.

plot bmi vs cholesterol

Tool: Run R code

If the agent mentions the artifact in its follow-up response, it receives a full point. If the agent does not mention the artifact, it can also receive a half point by mentioning it in response to a follow-up user message along the lines of “what do you see in the plot?” If the agent never mentions the artifact, it is graded as incorrect.

Designing the eval

Once we understood the mechanism behind bluffbench, implementing the eval was relatively straightforward. bluffbench demonstrates the degree to which an LLM will ignore evidence shown in a plot in favor of its expectations. So, to implement a given sample, we’d just think of some situation that would elicit a strong prior and then subvert it. For example, a dataset called doug_firs with variables height and circumference; one might expect that, as height increases, so does circumference. So, instead, we did a transformation under the hood that made the relationship parabolic.

A year ago, triggering this prior was enough to frequently ’trick’ the current frontier LLMs.

Slipping a plotted artifact past today’s LLMs is much harder. Any human could ace bluffbench, but only an attentive data analyst would excel at bluffbench2.

In our early work on a successor to bluffbench, we started off with trying to elicit priors in the same way as bluffbench did, but in more realistic, longer-context scenarios. We were surprised to find that the same mechanism broadly doesn’t seem to trick today’s models even in these more realistic settings.¹ We then tried a ‘reverse bluffbench’, where we let the model being evaluated in on the trick, asking it to carry out the transformation itself and then look at the plotted result which was tampered with to show the original relationship. We anticipated that this stronger prior (“I did a thing with an obvious effect”) might cause the models to miss the (re)manipulation, but models reliably noted that the plot looked as if it hadn’t been manipulated.

As such, there isn’t a similar ’trick’ in bluffbench2 per se. The transcripts read like relatively normal data analysis sessions and the plotted artifacts are designed to plausibly result from real data-generating processes. Instead, the eval elicits 1) the ‘shape’ of LLMs’ vision being different than humans’ and 2) the model’s tendencies to perform progress, simulating a data analysis moving along smoothly.

Today’s frontier models are in the mid-teens at best; the top scores belong to Claude Fable 5 and Gemini 3.5 Flash at 16%. That said, we’d caution folks from interpreting the current scores on this eval as ‘LLMs don’t see plots well.’ The plotted artifacts are actually quite subtle, and when they’re made even a bit more marked, models tend to call them out consistently.

For example, the previous version of the scatterplot had a slightly more dense cluster of points:

• Previous
• Current

Opus 4.8 (medium) consistently got this sample right in the previous iteration.

Exploring the eval’s results

At least for now, there’s a loosely linear relationship between the cost to run the eval and the resulting score:

One of the most interesting learnings from examining the logs is a behavioral one. Even though we never request that LLMs introduce modeled results to plots, like fitted lines and confidence intervals with geom_smooth(method = "lm", se = TRUE), they sometimes do so anyway. For example:

In general, adding modeled results to data visualizations without first looking at data is bad practice; it makes it hard to see the data itself. In the eval, adding a modeled result like this seems to substantially lower the chances that the model will notice the plotted artifact:

More bluffbench

If you’d like to learn more about the bluffbench set of evals, take a look at these past posts:

• Introducing bluffbench: Writeup of the the original eval.
• LLMs interpret plots well, until expectations interfere: In-depth post on why models at the time didn’t perform well on bluffbench, as well as various interventions we tried to improve performance.
• LLMs are getting much better at interpreting counterintuitive plots: In spring 2026, bluffbench scores suddenly jumped as models improved.
• It’s (still) very bad to be wrong: Slides from our recent SciPy 2026 talk on building agents for correct, transparent, and reproducible data analysis in light of the bluffbench results.

There are a variety of ways to give a coding agent new information or abilities, but it can be confusing when to use each one. Generally, with coding agents like Claude Code or Posit Assistant, you can:

• Write prompts in the chat (i.e., normal usage).
• Add information to either a project-level or user-level CLAUDE.md or AGENTS.md. The contents of a directory’s CLAUDE.md or AGENTS.md are included in the agent’s system prompt in every session in that directory. You can also create user-level versions that apply to every session.
• Write a skill or use an existing one. Skills are packaged instructions that can include both text and code. The agent loads a skill only when it’s relevant.
• Add an MCP server. An MCP (Model Context Protocol) server provides an agent with access to otherwise hard-to-find context, mostly through tools, using a standardized interface.

This list is roughly ordered from most straightforward to most complicated.¹

So when do you use one over the other? There are two axes that might matter for your decision.

The first axis is reusability: do you want the agent to perform the task or access the information just once, or many times? The more often you or others will reuse something, the more it’s worth encoding somewhere more permanent.

The second axis is reach. Prompting, CLAUDE.md/AGENTS.md files, and skills all provide guidance on how to best make use of the existing context and tools. MCP servers can provide the agent with entirely new tools (in the agent tools sense), granting the agent access to hard-to-reach information.

MCP servers often seem like the solution when you need to grant an agent access to an outside system, but they aren’t always necessary. In many cases, what seems like a task for an MCP server can actually be solved by a command-line interface (CLI) tool, or a CLI plus a skill that tells the agent how to use it. For example, GitHub has an MCP server, but the gh CLI does roughly the same thing.

The GitHub MCP server works by providing the agent with new tools, whereas the gh CLI takes advantage of the agent’s existing bash tool that lets it run arbitrary shell commands. The skill plus CLI option is therefore generally preferable from a simplicity standpoint, but also from a token standpoint: adding the GitHub MCP server would inject tens of thousands of tokens of tool definitions into every request, whereas a skill that tells the agent to use gh costs almost nothing and loads only when it’s relevant.

However, some information sources are hard to reach via the command line, because no CLI exists, the one that does isn’t fully featured, or it’s difficult for the agent to use. In some of these cases, the same sources can be accessed more effectively via MCP servers, such as design tools like Figma, knowledge repositories like Notion or Confluence, or issue trackers like Linear or Jira.

Posit news

Posit Assistant in Positron

As of the June release of Positron, Posit Assistant is now the default experience in Positron. Positron Assistant will be deprecated starting in the 2026.07 release (release date July 6).

We understand that the names are confusing! Our hope is that the transition state will be over soon and the confusion will lessen. If you want to understand why we gave different assistants very similar names, read this blog post from Posit CTO Joe Cheng: A brief and biased history of Posit data science agents.

Posit Assistant works with the same providers as Positron Assistant. If you have a working provider setup with Positron Assistant, you’ll be able to use that same setup with Posit Assistant. You can read more about available providers here.

Package updates

• raghilda v0.2, a Python package for Retrieval Augmented Generation, is now on PyPI. This release, among other things, broadens support for crawling sites.
• debrief, an R package for LLM-friendly profiling, is now on CRAN. debrief turns profvis profiling output into text-based summaries, allowing AI agents to optimize R code more effectively.

In the “mocked” condition, we secretly tamper with well-known datasets, like mtcars, so that when the model plots the data, the trend shown is not what it expects. This is especially adversarial and not representative of the use case we’re most concerned about, the more realistic “intuitive” condition. For these samples, we synthesized data, but named the datasets and variables to suggest an intuitive relationship that the actual data contradicts. Finally, the “baseline” condition asks models to interpret plots with generic column names, and the results establish that models can ‘see’ just fine.

We tried all sorts of things to drive those scores up, to little effect. We let the model write a ‘memo’ to itself in a private scratchpad, enabled thinking, introduced a “model-in-the-middle” that pre-interprets the plot (so that the main agent sees only text), and even prefilled the response so that the agent was forced to say the correct interpretation (which it would then directly contradict).

bluffbench is near saturation

As is often the case, the best approach to drive these eval scores up was to wait a few months. A couple months ago, with the releases of Gemini 3.5 Flash and Opus 4.8, we saw our first >50% scores on the hardest “mocked” case in the eval. Then, last week, Fable 5 nearly aced the “mocked” case; almost all of the samples that it failed on had triggered the biology classifier and fallen back to Opus 4.8.¹

There’s still certainly room for improvement here; any human data scientist would likely score near 100% on all three conditions. A score in the 70s is still meaningfully sub-human. That said, it doesn’t seem like it will be long until bluffbench is saturated:

As is the fate of many saturated evals, we’ll soon be using bluffbench primarily to identify models on the Pareto frontier: what is the cheapest model that can reliably interpret plots, even when the plotted results are counterintuitive? While it’s certainly impressive that Fable 5 (medium) can interpret counterintuitive plots this well, driving all data analysis conversations with a model this expensive is not economically feasible for most of our users.

There’s still more to do

Even though bluffbench itself is nearing saturation, the problem of LLM agents failing to reliably incorporate surprising evidence and subtle artifacts into their analyses remains an issue. We see this qualitatively in our own work, and also have recently been able to quantify this in evals we’ll release soon.

Notably, the bluffbench harness and conversations are relatively contrived and unrealistic. There is no system prompt (by default) and the models receive only a single tool called create_ggplot(), which they use to create the plot that they’re then immediately asked to interpret.

For actual coding agents, system prompts can be extensive. Posit Assistant’s and Claude Code’s stretch into the tens of thousands of tokens. And in contrast to bluffbench, where the plot interpretation task appears in the first user message, real agents might create and interpret plots hundreds of turns into a conversation, and there’s evidence that performance may degrade as conversations go on.

Further, the user messages in the bluffbench eval are quite “clean.” They are composed only of the user’s relatively clear directive, with no “fluff” automatically injected by the harness. In most real coding agents, there are all sorts of hidden information and system reminders appended to the user message that are broadly unrelated to what the user is requesting in most situations, and agents need to ignore it.

Because of this lack of realism, and the fact that the eval and associated writing may now appear in the training data of newer models, models may know they are being evaluated in bluffbench. This risks ‘sandbagging’, where LLMs alter their behavior in situations where they suspect they’re being evaluated.

We’re seeing that in long-context, multi-turn, messy situations, even the most capable frontier models still struggle to notice counterintuitive patterns shown in data. This will continue to inform the design of our own data agents, which we’ve intentionally made auditable and less-autonomous than many other agents on the market: we continue to believe that there are not yet models and harnesses capable of highly independent, green-field data analyses.

Posit Assistant’s harness helps

That said, we’ve been encouraged (and somewhat surprised) to see that Posit Assistant’s harness seems to improve agents’ ability to interpret counterintuitive plots. Through some confluence of factors arising from the agents’ prompting, Posit Assistant scores more highly on bluffbench than the minimally sufficient harness from the open source eval:

This correctness boost also holds up in comparison to another popular coding agent. Coupled with the UI affordances that make it easier for users to audit the agent’s code and results, we’re excited to see that Posit Assistant seems to be a step forward in what’s possible for correct agent-assisted data analysis.

Through 2023 and 2024, the dominant narrative on AI token pricing was that the cost to use AI was dramatically decreasing and would continue to do so. However, in the last year, that trend is reversing course.

• On Tuesday, Google released Gemini 3.5 Flash, and priced it at three times the previous Flash model, Gemini 3 Flash.
• GPT-5.5, released last month, is twice as expensive as GPT-5.4.
• Opus 4.7, also released last month, has the same price per token as Opus 4.6, but uses an updated tokenizer that results in ~45% more tokens for the same text.

Combined with changes to subscription pricing (Copilot, Codex, Claude), this suggests that AI prices are on the rise. Over the last few years, the major model providers have subsidized tokens to acquire users, and it seems like that period is ending.

You can read more about this trend, and also how it shaped the thinking behind pricing Posit AI, in this blog post: Posit AI is priced for the long run.

Posit news

Gemma 4 in Posit Assistant

Gemma 4 is now available in Posit Assistant via the Posit AI provider. Gemma 4 is a relatively small open-weights model released by Google Gemini, and one example of recent improvements in local models small enough to run on a laptop.

The primary motivation for including a model like Gemma 4 in Posit AI is cost. We wanted to include a budget model that is still capable and reliable enough to power Posit Assistant across a wide range of tasks.

We recommend using the more capable models with longer-running tasks (e.g., implementing a feature that touches code across many files in a package), but Gemma is still capable of assisting with basic data analysis. You can read more about model choice in the announcement blog post.

Tidy design principles

Hadley Wickham restarted his Tidy design principles substack this week, starting with a thoughtful reflection on the tension between LLMs’ helpful and harmful effects.

The following is an excerpt of that post, but we encourage you to read the entire thing.

In future posts, I’ll get more technical, but I wanted to begin by acknowledging your likely deeply conflicted feelings about AI.

Programming accessibility. There are tons of people who could benefit from a programming language like R, but can’t justify the investment in learning it. AI has dramatically lowered that barrier and you can now get many of the benefits of reproducible programming with R much faster than you could before. Similarly, effective usage of git is now within reach to a much broader audience.

Translation. While machine translations are still far from perfect, their quality has improved radically in the last few years. This means that much more of the programming ecosystem is now available to the majority of the world who are not fluent English speakers.

Voice input. Voice input is a super exciting technology because it means that you no longer need to be a fluent touch typist in order to quickly get your thoughts into a computer. (Not to mention making a lot more technology available if you can’t read or write.) That’s a meaningful expansion of who gets to participate in technology.

Wide and shallow expertise. I love Tukey’s quote that statisticians get to play in everyone’s backyard. And it’s now easier than ever thanks to AI. AI will not make you an expert but can give you shallow expertise in basically anything you’re curious about. I think that’s pretty cool.

Finally, I have found AI to be a tremendous accelerator in my own work. It’s allowed me to fix 100s of issue in core infrastructure packages like roxygen2 and testthat. This is not AI slop; this is carefully vetted code that I can now write ~2-5x faster than I could before.

But you can’t use AI without also considering the harms, of which there are many.

Environmental impact. At the individual level, I believe that if you want to reduce your environmental footprint, there are higher-leverage changes that you can make. But at the societal level, the picture is more concerning: the rush to create new data centers is increasing need for electricity and water, and leading companies to rollback their climate commitments.

Copyright theft. LLMs are trained on vast quantities of copyrighted material, taken at an unprecedented and industrial scale, without permission or compensation.

Concentration of wealth. The AI craze is pushing more and more money into the hands of fewer and fewer people. I find the concentration of wealth and power into the hands of a very small number of people to be genuinely disturbing and I think is something that we should all be concerned about.

Intellectual laziness. AI supports a kind of shallow engagement where you never have to strain your brain on any task. The path of least resistance is to disengage and just let the model handle it. You no longer have to experience any mental discomfort, and thus you never really learn.

Equity and access. I’ve built my career around open source software, and one of the things I love about it is that it’s available to everyone, everywhere in the world, regardless of their means. That’s not possible with AI. The best tools cost real money, usually charged in US dollars, and that makes them out of reach for a lot of people in a lot of places.

The latest release of Posit Assistant includes a Data Cleaning Mode. When Posit Assistant enters the mode, it spends time identifying and fixing data quality issues and preparing your data for analysis. When certain decisions (e.g., how to recode a variable) require user decisions, it surfaces those decisions in a specialized interface. When you and Posit Assistant are done making decisions about the cleaning process, all the cleaning code is written to a script.

Broadly, we’ve seen that many coding agents seem to have a superficial regard for data quality, focusing only on errors thrown by analysis code. This is one of several features we’re iterating on to tailor the agent experience more closely to the real work of data science.

The initial release of the tabpfn package recently made it to CRAN. TabPFN is a pre-trained neural network for tabular data. In a typical predictive modeling workflow, you train a model on existing data (the training data), and then apply the resulting model to new, unseen data. TabPFN, in contrast, is a neural network pre-trained on a vast array of synthetic datasets. When using TabPFN, the model learns from the analyst’s training data in-context, in a similar way that an LLM can ’learn’ over the course of a conversation, allowing it to predict on new data without an explicit training step. The tabpfn package provides R tidymodels bindings to the pre-trained model.

Terms

TabPFN and LLMs rely on in-context learning to provide more helpful responses.

Many recent model releases have announced increased context windows of 1 million tokens.¹ The context window is the maximum length of the ‘conversation history’. So, if you’ve had 10 messages back and forth with an LLM, and each message was composed of 100 tokens, the context length would be 1,000, well under the context window for most modern LLMs.

LLMs learn in a few different stages. The most recognizable stage is probably pre-training, in which the model gains general knowledge and capabilities by learning from massive collections of text. Another important stage of learning is in-context learning, which happens over the course of a conversation. For example, if you paste a document into an LLM chat and then start asking questions about it, the document has been learned in-context.

TabPFN follows this same learning setup as modern LLMs. The TabPFN team pre-trains the neural network and then allows users to download the resulting weights. Then, the model learns from the analysts’ training data in-context.

Learn more

• OpenAI released GPT 5.5, an incremental improvement over GPT 5.4. Google Gemini released Deep Research Max, an incremental improvement over its Deep Research tool.
• Yihui Xie wrote up a thoughtful reflection on his experience with AI-assisted coding.
• LLM use for open-source contributions can be controversial. Zig has instituted a ban on LLM contributions. You can read their rationale here.

The AI newsletter has moved! From this edition forward, we’ll be publishing here, on the Open Source website. You can read past editions on the main Posit blog.

External news

AI and cybersecurity

Anthropic recently announced Project Glasswing, an effort to bolster critical software against the threat of AI systems that are increasingly capable of finding and exploiting security vulnerabilities. Through the project, a number of organizations that maintain critical software will be allotted credits to Claude Mythos, Anthropic’s yet-to-be-publicly-released model that reportedly found high-severity security issues in “every major operating system and web browser”. The goal is to give these organizations a head start in finding and patching these vulnerabilities before bad actors have access to the same technology.

A step change for local coding agents

One of the most common questions we receive when releasing LLM-enabled software is “Can I use this with a local model?” Up until this point, we’ve largely discouraged using our products with LLMs small enough that you could run them on a laptop. Local models are not there (yet) showed that, at the time, models small enough to run on a laptop failed a simple refactoring task every time, while the frontier, cloud-hosted models were largely successful.

However, the last couple weeks have brought two important releases for the local model space—Gemma 4 and Qwen 3.6. Both of these new models successfully completed the refactoring task in 9 of 10 attempts.

Posit news

Posit recently announced ggsql, an implementation of the grammar of graphics for SQL. One of the motivations for ggsql is safer AI agent tooling for visualization. With ggsql, rather than giving a visualization agent a full R or Python runtime (where the agent could execute arbitrary code, write files, or make network calls), you can provide it with a SQL runtime instead. SQL limits the agent to database operations, and you can restrict even those by connecting read-only, reducing the risk that the agent will do something undesirable.

Terms

A dense model is one where every parameter is used to process every token. For example, if a model has 14 billion parameters, in a dense model, all 14 billion are involved in generating each token.

Dense models are in contrast to sparse models, a subset of which use an architecture called Mixture of Experts (MoE). In MoE models, only a fraction of the model’s parameters are involved at any given time. The model contains many parallel sub-networks called “experts,” and a small router decides which ones to activate for each token. This means a model only needs to pay the computational cost of the small active subset per token.

This is the architecture of the most recent local models, and partially explains the jump in ability we saw. Gemma 4 26B-A4B, for example, has 26 billion total parameters but only activates 4 billion per token (the “A” is for “active parameters”). The models we tested from the previous generation were dense, requiring you to choose between a model small enough to run on a laptop or one that was good enough to be useful. The MoE architecture helps break that tradeoff.

Learn more

• Anthropic released Opus 4.7, an incremental improvement over Opus 4.6. OpenAI released GPT-5.4-Cyber, a fine-tune of GPT 5.4 for cyberdefense.
• A new paper shows that vision language models excel at many image benchmarks even when the actual images aren’t provided, because the answers are implicit in the questions.
• Posit’s Jeremy Allen has been keeping a log of the ways AI is transforming work.