Week 6: From Blueprint to Code

Overview

This week I rewrote the entire archive carousel in TypeScript using PuerTS (a plugin that runs TypeScript/JavaScript inside Unreal Engine). The Blueprint version worked, but every change required dragging nodes and reconnecting wires. The TypeScript version gives me a code-first workflow where I can iterate on layout math, opacity curves, and interaction logic in seconds instead of minutes.

Goals This Week

Why PuerTS

The Blueprint carousel (BP_ArchiveCarousel) worked, but it had real limitations:

Math is painful in BP: calculating sphere positions, wrapping angles, opacity curves, all through visual nodes. A single formula change meant rewiring 10+ nodes.
Iteration speed: every tweak required navigating the Blueprint editor, finding the right node, changing a value, compiling. In TypeScript, I change a number and recompile in under a second.
Version control: Blueprint is binary. I can’t diff it, can’t review it, can’t track what changed. TypeScript is plain text.

PuerTS lets me write TypeScript that compiles to JavaScript and runs inside UE5. The actor lifecycle (BeginPlay, Tick, EndPlay) maps directly to class methods. UE APIs are available through typed bindings.

The tradeoff: PuerTS doesn’t expose everything. Enhanced Input’s BindAction isn’t available, so gesture input still routes through Blueprint. The pattern becomes: Blueprint handles detection, TypeScript handles logic.

The Rebuild

Sphere Layout

The carousel places 18 cards (3 rings of 6) on a 180-degree spherical arc. Each ring sits at a different height, and the sphere geometry naturally tapers the radius at top and bottom rings, giving the layout a curved screen feel.

Key parameters, all tunable in code:

CarouselRadius: 80 cm
RingVerticalSpacing: 30 cm
ArcDegrees: 180 degrees
CardDrawSize: 640 x 360 (matching the WBP_CarouselCard SizeBox)

Wrapping

Cards wrap both horizontally and vertically. Scroll past the edge and cards reappear on the other side. This creates the illusion of an infinite archive from a finite set of cards.

Opacity and Scale

Cards at the center are fully opaque and full size. Moving toward the edges, they fade to 70% opacity and shrink to 90% scale. Near the wrap boundary, they fade to zero and become invisible. This creates depth without requiring a depth-of-field shader.

The opacity implementation was one of the harder problems. In Blueprint, you’d set material opacity. But WidgetComponents don’t use materials. After testing multiple approaches (SetTintColorAndOpacity, SetColorAndOpacity, SetRenderOpacity on the UserWidget), the solution was: cache each card’s root SizeBox widget via WidgetTree.RootWidget and call SetRenderOpacity on that. The root widget isn’t available immediately after spawn, so the code waits 30 frames before caching.

Zoom

The carousel supports dynamic zoom from 1x to 5x. Zooming scales the radius, vertical spacing, and card size together, so the whole structure grows uniformly. Keyboard controls (U/O) work for desktop testing. For VR, there’s a pinch-to-zoom API: Blueprint detects two-hand pinch and sends fingertip distance each frame; TypeScript calculates the zoom ratio from the initial pinch distance.

Input Architecture

PuerTS can’t bind Enhanced Input Actions directly. The solution: TypeScript exposes BP-callable functions, and Blueprint’s Event Graph calls them.

Gesture	Blueprint Detects	TypeScript Function
Swipe Left	IA_MG_L_LeftSwipe	SwipeLeft()
Swipe Right	IA_MG_L_RightSwipe	SwipeRight()
Swipe Up	IA_MG_L_ForwardSwipe	SwipeUp()
Swipe Down	IA_MG_L_BackwardSwipe	SwipeDown()
Pinch Zoom	Both hands pinch + distance	UpdatePinchZoom(dist) / EndPinchZoom()

Card interaction (click, hover) is handled entirely in TypeScript through WBP_CarouselCard’s event dispatchers, which PuerTS can bind to via the Add() method on multicast delegates.

PuerTS Lessons Learned

Things that aren’t obvious from the documentation:

Always compile manually: npx tsc after every change. There’s no hot reload. Forgetting this caused hours of debugging “changes not working.”
Initialize in BeginPlay, not Constructor: PuerTS clears property values during UE object construction. All initialization must happen in ReceiveBeginPlay().
Widget roots need delay: WidgetTree.RootWidget returns null immediately after widget creation. Wait ~30 frames before caching.
BP variables are accessible: (widget as any).TXT_Location works for accessing Blueprint-exposed variables on widget instances.
SlateColor needs two arguments: new UE.SlateColor(color, ESlateColorStylingMode.UseColor_Specified), not just the color.

ISDK Migration

Mid-week, I migrated the WidgetComponents to Meta’s Interaction SDK (IsdkInteractableWidgetComponent). This adds native ray and poke interactable support without manually configuring collision volumes.

Key details:

WidgetScale defaults to 0.02, which makes cards nearly invisible. Must set to 1.0 explicitly.
Rounded box material: bUseRoundedBoxMaterial=true with CornerRadius=(30,30,30,30). BlendMode must be Transparent (Masked causes aliasing at edges).
PointablePlane sizing: default 20x20 is far too small for the cards. Must set to CardDrawWidth * BaseWorldScale * 0.85 to match card world size.
Poke doesn’t work with manually created widgets via SetWidget (known ISDK limitation). Ray + pinch works perfectly. ThumbTap works through BP-callable bridge.
Click deduplication: 0.2s same-card filter using GetRealTimeSeconds() to prevent double triggers.

Photo Loading and Level Streaming

21 world photos loaded into the carousel with location labels (“Glacier, Iceland”, “Ama, Tainan”, etc.). Each card maps to a 3DGS level.

The full interaction flow: card click triggers HideCarousel(), pawn fades out, the current level unloads and the new 3DGS level streams in, then pawn fades back in. Uses Unreal’s LatentActionInfo callback pattern for async level load/unload sequencing.

Virtual scrolling expanded from 3 rings to 7 data rows, so only 3 rings of cards are visible at a time but the user can scroll through all 21 entries with wrap-around.

The carousel now snaps to the player’s headset position when opened. It calculates the HMD forward direction and places the carousel centered in front of the viewer. Once open, the carousel stays in place so hand interaction feels stable.

Toggle is wired through a palm menu: WBP_PalmMenu fires an Event Dispatcher on the pawn, which calls ToggleCarousel() on TS_WidgetCarousel.

Tab System and Dynamic Layout

Late in the week, I added a tab system to organize the archive into three categories: Travel (15 photos), Personal (3), and Dynamic (1, for 4DGS sequences).

Instead of visible tab buttons, the interaction is purely gestural: swipe up/down switches between tabs. A centered label appears briefly showing the current tab name (uppercase, bold), then fades out over 3 seconds. This keeps the UI clean while still giving feedback on which category is active.

Each tab has a different number of items, so the layout adapts dynamically. RecalcLayout() adjusts ring count, items per ring, and arc degrees based on the data size. Small tabs (1-3 items) use a single ring. Medium tabs use 2-3 rings. The arc narrows for fewer items (~30 degrees per card) so cards don’t spread out across the full 180 degrees. Vertical scrolling is disabled when all rings are visible.

The tab switch includes a slide + fade animation: cards fade out while sliding vertically, content swaps at the midpoint, then new cards fade in from the opposite direction.

Other changes: switched cards and tabs from Transparent to Masked blend mode. Transparent blend causes backgrounds to be invisible because the ISDK rounded box material multiplies with the widget background color. Masked makes backgrounds actually render. Also removed the vertical opacity gradient, so all rings now display at uniform brightness.

GS Fade System

When the carousel opens, the 3DGS scene dims so the cards are easier to read. When it closes, the scene brightens back. The system finds all GS actors in the level via duck typing (checking for SetBrightness on every actor) and controls their brightness and audio volume through per-frame lerping.

Transition	Target Brightness	Speed	Duration
Level load fade-in	0 to 1.0	0.33/s	~3s
Carousel open	1.0 to 0.1	0.6/s	~1.5s
Carousel close	0.1 to 1.0	0.6/s	~1.5s

Audio volume is capped at brightness * 0.6 to prevent full-blast audio even at max brightness. GS actors must have their default Brightness and AudioVolume set to 0 in the editor to prevent a bright flash on level load before the fade-in kicks in.

Onboarding Flow

The day before Demo Day, I built a full onboarding sequence: a 4-step walkthrough that introduces the viewer to the experience before the carousel appears.

Steps and Timing

Welcome (6s hold): “Hey, welcome. I’m Dazai.” followed by “This is my spatial memory album, a place to share my experiences in life.”
Palm Tutorial (7s hold): “Look at your palm to open the menu.” with a looping video showing the palm gesture.
Swipe Tutorial (9s hold): “Swipe your thumb left / right to browse” with a left-right swipe video. After 5 seconds, a second panel appears: “Swipe your thumb up / down to switch albums” with an up-down swipe video.
Farewell (6s hold): “Take your time, look around. Each place here holds a little piece of my story.”

Each step fades in over 0.8s, holds, then fades out over 0.8s before the next step begins. A 1.5s delay before the first step gives the viewer time to settle into the headset.

Video Tutorials

Each tutorial step plays a looping MP4 video inside a UMG Image widget via UE’s MediaPlayer/MediaTexture pipeline. The key discovery: SetBrushFromTexture crashes with MediaTexture. Instead, you have to assign the texture through brush.ResourceObject = mediaTexture directly. MediaPlayer opens the source and loops on step entry, closes on step exit.

Voiceover

Each step has narration recorded through ElevenLabs TTS, split from a single recording into 4 clips using ffmpeg’s silence detection (silencedetect=noise=-25dB:d=0.3). The audio plays via SpawnSound2D (not PlaySound2D) because SpawnSound2D returns an AudioComponent handle that can be faded out on skip.

VR Positioning

The onboarding widgets snap to the headset position every frame during the delay period, so they’re always centered in front of the viewer regardless of head movement. The position tracks the HMD’s forward direction at distance 120 with a Z offset of -30.

Skip Button

A skip button (ISDK ray-clickable) sits in the top-right corner. Pressing it fades out the current audio over 0.5s and jumps straight to revealing the carousel.

Carousel Integration

During onboarding, the carousel is locked (bLocked = true) to prevent accidental palm-menu toggles. Collision is disabled on the carousel actor when hidden. On onboarding completion or skip, the carousel unlocks, enables collision, resets to the Travel tab, and fades in over 2 seconds.

Polish Details

Card dates: each card now displays a date string (e.g., “2024.09.12”) via the TXT_Date text block in WBP_CarouselCard.
Tab rename: “Dynamic” renamed to “Motion” for clarity.
Tab animation speed: increased to 4x (exit + enter), making tab switches feel snappier at ~0.5s total.
Tab label color: fades from white to black instead of using opacity, preventing over-exposure on the translucent background.
Font: switched to Montserrat across all onboarding widgets for a cleaner look.
Desktop testing keys: N/M cycle card selection, Enter clicks, T toggles carousel, Space skips onboarding, I/K switch tabs.

Testing the completed widget carousel: scrolling through photo cards, selecting entries, and loading different 3DGS levels.

Full Session Recording (March 10)

Full development session recording covering onboarding flow, voiceover integration, video tutorials, and carousel polish: YouTube

Resolved Questions

Entering a space from the carousel: Card click triggers level streaming with fade transitions. Working end-to-end.
Pinch-to-zoom wiring: Lives in BP_MRPawn, using IsdkHandRigComponent to get fingertip positions from both hands.
Preview textures: Pre-loaded at spawn. 21 photos at 640x360 is lightweight enough to load upfront.
First-time user guidance: Onboarding flow with video tutorials and voiceover walks new users through gestures before showing the carousel.
GS scene distraction: Brightness fade dims the 3DGS scene when the carousel is open, keeping focus on the cards.

Open Questions

Full VR headset testing with all 21 levels
Meta Quest system gesture cannot be disabled (no official method as of v81+)

What’s Next

Post Demo Day

Iterate based on Demo Day feedback
Return-to-archive flow refinement
Full VR stress test across all levels

Week 6 | 2026-03-03 ~ 2026-03-11