Shipping AR Try-On to React Native and Expo Go with a WebView Engine

Once the web version of TryOnIt worked, the next question was obvious: can a React Native app use the same products? Most of my day-to-day work is React Native, so I wanted the answer to be “yes, today”, not “after a rewrite”.

There were two ways to get there. A fully native engine would give the best possible performance, but it needs native code in Swift, Kotlin and C++, a custom development build, and a renderer port. Running the existing web engine inside a WebView would work immediately, including in Expo Go, with exactly the same behaviour as the web.

I shipped the WebView engine first, and designed the public API so a native engine can replace it later without apps changing a line. This article explains both halves of that decision.

What a fully native engine needs

The native design is worth describing first, because it explains why it is not version 0.1:

react-native-vision-camera v5 (Nitro Modules)
  frame processor plugin (Nitro, C++ / Swift / Kotlin)
    Android: MediaPipe Tasks Vision (face, hand, pose, segmenter)
    iOS:     MediaPipeTasksVision (CocoaPods or SPM)
      returns landmarks and matrices (small arrays) to JS worklets
        @tryonit/core anchors and One Euro smoothing

Rendering
  makeup, hair, 2D stickers: React Native Skia (GLSL ported to SkSL runtime effects)
  3D products:               Filament (react-native-filament) with the same GLB files

The goal of that design is that camera frames never cross the JavaScript bridge as pixels. Only small arrays of landmarks reach JavaScript. The web makeup shaders, a luminance-aware composite and a separable blur, map directly to Skia’s RuntimeEffect language, and Filament can render the same GLB models with the same millimetre and axis conventions, with occluders becoming depth-only materials.

It is the right long-term design, but it requires a development build (it cannot run in Expo Go), an Expo config plugin that adds the MediaPipe dependencies, and a native plugin per platform. That is a lot of surface area to stabilise before anyone can try a lipstick.

Phase 1: the web engine inside a WebView

@tryonit/react-native 0.1 hosts the browser engine inside react-native-webview, which Expo Go already includes. The pieces fit together like this:

React Native app
  TryOnButton / TryOnView / TryOnModal / useTryOn
    props, ref API and events
      react-native-webview, page served on https://tryonit.local/
        TryOnIt core, web engine and message bridge (embedded at build time)
        MediaPipe (loaded lazily from a CDN through an import map)
        three.js (loaded lazily, 3D products only)

A few details make this work well:

• The engine is embedded in the package. A build script bundles the core, the web engine and a message bridge into the published package, so the app does not need to host any TryOnIt code.
• The page runs on a secure origin. Browsers only allow the camera in a secure context, so the page is served on https://tryonit.local/ inside the WebView.
• Heavy dependencies stay lazy. MediaPipe and three.js load from a CDN through an import map the first time they are needed, three.js only for 3D products, and the WebView caches them afterwards.
• The bridge is typed. Commands go into the page through injectJavaScript, and events come back through postMessage, both defined by one protocol module. Only these small messages cross the bridge; the camera stream stays inside the page.
• Products are validated before they reach the WebView. Manifests are resolved and validated with @tryonit/core on the React Native side, so invalid products fail with readable errors in your app, not silently inside a web page.

The core made reuse cheap

This only worked because @tryonit/core was platform agnostic from the start. It compiles without DOM types, never touches window, document or navigator, resolves URLs without the URL global, and schedules store updates with queueMicrotask and a Promise fallback, both available in Hermes. A test imports the whole public API in Node without DOM globals.

So these pieces are shared between web and native without modification: validateManifest, createSessionStore with the same session states, the face, hand and body anchor functions, estimateMetricScale, the One Euro filters, getAssetRequirements and the error codes. Theme token names and labels are identical too, so a brand’s web and app experiences look and read the same. The reasoning behind that core is in the architecture article.

Using it in an app

In Expo, install the package with the WebView:

npx expo install @tryonit/react-native react-native-webview

Expo Go already includes the WebView and the camera permission, so npx expo start is enough to try it. For development and store builds, add the config plugin so the permission reaches the native projects:

{
  "expo": {
    "plugins": [
      [
        "@tryonit/react-native",
        { "cameraPermission": "Allow $(PRODUCT_NAME) to use the camera so you can try products on." }
      ]
    ]
  }
}

The drop-in button asks for camera permission on Android, then opens a full-screen modal (a page sheet on iOS) with the camera, a shutter, a camera switch, a photo upload fallback and status messages:

import { TryOnButton } from '@tryonit/react-native';

export function ProductScreen() {
  return (
    <TryOnButton
      asset="https://cdn.example.com/tryon/aviator.json"
      title="Aviator Sunglasses"
      theme={{ colors: { primary: '#0f766e' }, radius: 12 }}
      onCapture={(photo) => console.log(photo.width, photo.height)}
      onError={(error) => console.warn(error.code, error.message)}
    />
  );
}

For a custom interface, useTryOn gives you the same controls as the web hooks and a camera-only view:

import { Button, View } from 'react-native';
import { TryOnView, useTryOn } from '@tryonit/react-native';

const lipstick = {
  version: 1,
  id: 'ruby',
  type: 'makeup.lips',
  color: '#B0123A',
  variants: [
    { id: 'ruby', name: 'Ruby', swatch: '#B0123A' },
    { id: 'nude', name: 'Nude', swatch: '#C48A7A', overrides: { color: '#C48A7A' } },
  ],
} as const;

export function CustomTryOn() {
  const tryOn = useTryOn();
  return (
    <View style={{ flex: 1 }}>
      <TryOnView {...tryOn.viewProps} controls="none" asset={lipstick} />
      <Button title="Nude" onPress={() => tryOn.setVariant('nude')} />
      <Button title="Flip camera" onPress={() => tryOn.switchCamera()} />
    </View>
  );
}

Captured photos come back as { dataUrl, base64, mimeType, width, height }. The dataUrl works directly in <Image source={{ uri }} />, and the base64 string can be written to a file and shared with Expo’s file system and sharing modules.

One React Native difference catches people out: relative URLs like /assets/lipstick.json do not exist in an app. Pass a manifest object, an absolute https:// URL or an async loader. Files referenced inside a manifest loaded from a URL may stay relative to it, but they must be hosted on a CDN with CORS enabled, because the camera page cannot read files bundled in the app. The manifest format itself is the same as on the web; see How to Create Virtual Try-On Products with JSON Manifests.

Requirements and trade-offs

The WebView approach sets the minimum platform versions:

Platform	Minimum
iOS	16.4, for WebKit import maps and WebAssembly SIMD
Android	8.0 with an up to date Android System WebView (Chrome 111+)
React Native	0.73
Expo	SDK 50 (tested with SDK 57)
Network	Needed the first time, unless MediaPipe and models are self-hosted

The trade-off is honest: a WebView adds overhead compared to a native pipeline, and older phones benefit from engineOptions={{ performance: 'battery' }}. In return, the package works everywhere React Native runs, including Expo Go, and shares 100 percent of its behaviour with the web packages. Every fix to the web engine is automatically a fix on mobile.

How it is tested

Two layers of tests cover the package. An end-to-end Playwright test loads the exact WebView page in Chromium with a fake camera, which exercises the engine, the bridge protocol and the UI without a phone. Separately, the Expo example app is bundled with Metro for both iOS and Android to catch packaging and resolution problems.

The road to a native engine

The public API was designed so the native engine can slot in behind it. The plan is incremental:

1. WebView engine (shipped in 0.1).
2. A Nitro frame processor plugin for the face landmarker on iOS and Android, with anchors computed by the shared core.
3. A Skia makeup renderer, lips first and then every makeup type.
4. A Filament 3D renderer, starting with glasses and the head occluder.
5. Hand tracking for watches and rings, hair segmentation, and pose tracking for clothing.
6. Components that mirror the web API: TryOnButton, TryOn, useTryOn and useTryOnState.

Apps that adopt the WebView version today should be able to upgrade without code changes. The package is on npm as @tryonit/react-native.