Charlie Gao

Senior Software Engineer at Posit, PBC

shikokuchuo.net

shikokuchuo

charliegao

shikokuchuo

0000-0002-0750-061X

Software: mirai mori nanonext Quarto Shiny purrr promises mcptools

Charlie Gao is an open source software engineer at Posit building asynchronous and parallel tooling in C, R and Python for data science, scientific computing, and technical communication across Tidyverse, Shiny and Quarto.

He is the author of mirai, a high-performance framework for async, parallel, and distributed computing in R, and mizu, lock-free shared-memory parallelism for R and Python built on a C core. He is also part of the Quarto core team developing Quarto 2, with a focus on CRDT-based real-time collaboration.

Software by Charlie Gao#

Events attended by Charlie Gao#

Posts and resources by Charlie Gao#

Building Reproducible Pipelines in R | Will Landau | Data Science Hangout

ADD THE DATA SCIENCE HANGOUT TO YOUR CALENDAR HERE: https://pos.it/dsh - All are welcome! We’d love to see you! This week’s guest was Will Landau, Senior Advisor of Innovative Statistics at Eli Lilly! Some topics covered in this week’s Hangout were building reproducible pipelines with the {targets} package, distributed and parallel computing with {crew} and {mirai}, designing trustworthy AI agents, choosing between Bayesian and frequentist approaches in clinical trials, and maintaining open source software in the age of AI. One community member asked: When do you reach for a Bayesian approach over a frequentist one in a clinical trial, and how do you explain that choice to stakeholders? Will’s paraphrased answer: The familiar reason is prior information, whether you want to be opinionated about incorporating it through a rigorous prior or techniques like historical borrowing. But he’s implemented plenty of Bayesian models on diffuse priors too, because they make early-phase decisions much easier to quantify. Rather than only asking whether a result is statistically significant, Bayesian models let you more conveniently answer the practical question: for example, the posterior probability of observing a large enough effect size, which is incredibly useful when deciding whether to move a therapy to a large Phase 3 registration study. You also don’t need frequentist asymptotic approximations like the delta method, you can work directly with MCMC (markov chain monte carlo) and transformations on posterior samples. Resources mentioned in the video and chat: targets Package book→ https://books.ropensci.org/targets/ targets Package Documentation → https://docs.ropensci.org/targets/ targets on GitHub → https://github.com/ropensci/targets crew Package (distributed computing) → https://github.com/wlandau/crew mirai Package (async, parallel, and distributed computing) → https://mirai.r-lib.org/ Trusted Mini-Agents Guide (Will Landau) → https://wlandau.github.io/trusted-mini-agents/ RTMB Package → https://cran.r-project.org/web/packages/RTMB/index.html Will Landau’s targets Workshop at R/Pharma 2023 → https://www.youtube.com/watch?v=cyF2dzloVLo&list=PLMtxz1fUYA5C0csy8-wTfFmRYQEB30vr3&index=12 Will Landau & Charlie Gao joint talk at posit::conf(2025) → https://www.youtube.com/watch?v=s_cBi155ZIQ Data Science Lab episode with Charlie Gao on mirai → https://www.youtube.com/watch?v=Vf_jmY1tMWo&list=PL9HYL-VRX0oSeWeMEGQt0id7adYQXebhT&index=1 “AI and the Invisible Newcomer in Open Source” by Mara Averick → https://blog.stdlib.io/ai-and-the-invisible-newcomer-in-open-source/ Kahn’s Algorithm for Topological Sorting → https://medium.com/@robhernandez5/kahns-algorithm-for-topological-sorting-26f29f2eaf48 XKCD: Dependency comic → https://xkcd.com/2347/ MLU-Explain (visual machine learning explanations) → https://mlu-explain.github.io/ Posit Academy → https://academy.posit.co/ ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co Hangout: https://pos.it/dsh The Lab: https://pos.it/dslab LinkedIn: https://www.linkedin.com/company/posit-software Bluesky: https://bsky.app/profile/posit.co Thanks for hanging out with us! Timestamps: 00:00 Introduction 04:58 “Can you talk about the targets package?” 13:21 “You’re doing great work to try to separate out that development support stuff from your day-to-day job and personal time and life. How the heck do you do it?” 16:57 “As someone who’s primarily worked with tools like SnakeMake and Nextflow, why should I use targets, or what are some benefits of using targets over something like Nextflow?” 20:37 “How has Gen AI affected you as a package author and maintainer, mostly from other people submitting issues or PRs, and less about how you use it personally in your own work?” 25:06 “Do you want to talk a little bit about integrating Mirai to do things even faster?” 29:10 “How do you communicate uncertainty to people who want a single number?” 31:24 “What does an average day or week look like for you? How is your time divided between development, consulting, etcetera?” 34:00 “Have you had feedback from people using targets for agentic LLM workflows, for testing parameterization changes of RAG ingestion and associated changes to how the LLM instructions are dealing with that, or just anything in that creating and testing agentic workflows space?” 37:15 “When do you reach for a Bayesian approach over a frequentist one in a clinical trial, and how do you explain that choice to stakeholders?” 41:32 “Which Bayesian MCMC tool do you use? Do you use Stan, or the macros to get to Stan?” 45:56 “Is there a piece of career advice that you wish you knew before you started in data science, or anything that you think would be helpful for everybody else here?”

Charlie Gao

Async & Parallel R with {mirai} | Charlie Gao | Data Science Lab

The Data Science Lab is a live weekly call. Register at pos.it/dslab! Discord invites go out each week on lives calls. We’d love to have you! The Lab is an open, messy space for learning and asking questions. Think of it like pair coding with a friend or two. Learn something new, and share what you know to help others grow. On this call, Libby Heeren is joined by Charlie Gao, who walks through async and parallel programming using the mirai package for R. Charlie is the author of {mirai} and {mori}! Charlie demonstrates how mirai lets you run R code in parallel across multiple cores or even distributed across remote machines. He covers the key differences between parallel and async programming, shows how to set up worker daemons (said like “demons”), scale resources dynamically, and connect to remote machines via SSH. Whether you’re running long computations, training models, or building Shiny apps, mirai helps you make the most of your computing resources without blocking your main R session:) Hosting crew from Posit: Libby Heeren, Isabella Velasquez Charlie Gao’s GitHub: https://github.com/shikokuchuo Charlie Gao’s Bluesky: https://bsky.app/profile/shikokuchuo.net Charlie Gao’s LinkedIn: https://www.linkedin.com/in/charliegao/ Charlie Gao’s Mastodon: https://fosstodon.org/@shikokuchuo Resources mentioned in the video and chat: mirai package website: https://mirai.r-lib.org/ mirai GitHub repository: https://github.com/r-lib/mirai AskDeepSeek chatbot for mirai documentation: https://mirai.r-lib.org/ (click “ask deep wiki” button) mirai - Promises (Shiny and Plumber): https://mirai.r-lib.org/articles/v02-promises.html mirai - Serialization: https://mirai.r-lib.org/articles/v03-serialization.html mirai - OpenTelemetry: https://mirai.r-lib.org/articles/v05-opentelemetry.html mirai stop_mirai function: https://mirai.r-lib.org/reference/stop_mirai.html mirai skill for Claude Code: https://github.com/r-lib/mirai/blob/main/.claude/skills/mirai/SKILL.md Plumber2 package: https://plumber2.posit.co/ Daemon (computing) on Wikipedia: https://en.wikipedia.org/wiki/Daemon_(computing) ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co Hangout: https://pos.it/dsh The Lab: https://pos.it/dslab LinkedIn: https://www.linkedin.com/company/posit-software Bluesky: https://bsky.app/profile/posit.co Thanks for hanging out with us! Timestamps of Questions / Topics: 00:00 Introduction 03:33 “What is async programming and how is it different from parallel?” 06:57 AskDeepSeek chatbot for the Mirai package 09:40 Setting up the Positron IDE with activity bar on top 15:25 “What was the motivation or need for developing Mirai?” 22:10 Mirai map function for parallel processing 25:25 “Do the contents of Mirai inherit definitions from the global environment?” 29:00 “What’s the difference between Mirai versus promises and future?” 31:50 Demonstrating sequential vs parallel processing 36:00 “Can you use Mirai with HPC?” 37:07 Dynamically scaling workers by adding and removing daemons 38:38 Setting up daemons with URLs for network connections 42:05 Launching workers over SSH to remote machines 43:09 SSH tunneling to connect workers without open ports 51:14 “Does Mirai allow R to perform the same thing as NumPy?” 51:54 “Does Mirai ship with some kind of task viewing dashboard?” 52:35 “Can we use parallel::detectCores() to see how many daemons we can use?” 53:58 “Would parallel processing be more useful than async processing in typical data science work?” 55:00 Mirai skill for Claude Code and AI agents

Charlie Gao

The Future of Asynchronous Programming in R - Charlie Gao

Asynchronous programming can be a powerful paradigm, whereby computations are allowed to run concurrently without blocking the main session. It is an opportune time to survey the current landscape, as R infrastructure in this respect has matured significantly over recent years. Instead of running a script sequentially from top to bottom, logic that takes a long or unpredictable amount of time to complete may be offloaded to different R processes, possibly on other computers or in the cloud. In the meantime, the main session may be running constantly and non-interactively, performing operations in real time, synchronizing with these tasks only when necessary. This style of programming requires a very specific set of tooling. At the very base, there is an infrastructure layer involving key enabling packages such as later and mirai. It will be explained at a high level why these two packages together currently offer the most complete and efficient implementation of async for the R language. There are further tools which expand async functionality to cover specific needs, such as the watcher package for filesystem monitoring. There are then a range of tools built on top of these, bringing async capabilities to the end-user, such as the httr2 package for querying APIs and the ellmer package for interacting with LLMs. In addition to these existing tools, exciting developments in asynchronous programming are just around the corner. These will be previewed, together with speculation on what might be possible at some point in the future

Charlie Gao