Tech Dev Log | Spatial Memory

3DGS Source

Source video: IMG_0922.MOV (iPhone)
Location: Brooklyn, NY apartment
Processing: Luma AI
Unreal project path: Content/Movies/IMG_0922.MOV

BP_Clip

Each BP_Clip instance = one memory fragment placed in the level.

Actor Structure

BP_Clip (Actor)
|
+-- Root (Scene Component)
|   = Memory location in the space (e.g., desk surface, window, kitchen counter)
|
+-- Plane (Static Mesh: Plane)
|   Child of Root
|   = The visible frame (video or photo)
|   Material: Dynamic Material Instance
|       Video: Media Texture from Media Player
|       Photo: Texture2D
|
+-- MediaSoundComponent
    = Audio output, linked to Media Player in BeginPlay

Variables (Instance Editable)

Variable	Type	Purpose
`bIsVideo`	Boolean	Photo or video clip
`MediaPlayer`	Media Player Object Ref	MP_ClipXX
`MediaSource`	Media Source Object Ref	MS_ClipXX
`MediaTexture`	Media Texture Object Ref	MT_ClipXX
`ClipTexture`	Texture2D	Photo texture (when not video)
`PlaneSize`	Float	Frame dimensions
`TriggerDistance`	Float	How close viewer needs to be
`ActivationAngle`	Float	Outer threshold in degrees
`FullRevealAngle`	Float	Inner threshold in degrees

Blueprint Logic

Construction Script:

Create Dynamic Material Instance → SET DynMaterial
  → Branch: bIsVideo?
      True:  Set Texture Parameter Value ("BaseTexture", MediaTexture)
      False: Set Texture Parameter Value ("BaseTexture", ClipTexture)

BeginPlay:

Branch: bIsVideo?
  True:
    → MediaSound → Set Media Player (MediaPlayer)
    → MediaPlayer → Open Source (MediaSource)
  False:
    → (nothing)

Note: Play on Open enabled on Media Player asset. No need to call Play in BeginPlay.

Event Tick:

1. Gaze Detection:
   distFactor = remap Distance(Player, Root) from TriggerDistance..0 → 0..1
   angleFactor = remap Dot(PlayerForward, DirToPlane) from cos(ActivationAngle)..1 → 0..1
   openFactor = distFactor * angleFactor

2. Smooth lerp:
   currentOpen = FInterpTo(currentOpen, openFactor, DeltaTime, smoothing)

3. Set opacity:
   DynMaterial → Set Scalar Parameter Value ("Opacity", currentOpen)

4. Video control (if bIsVideo):
   currentOpen > 0.1 AND not playing → Play
   currentOpen <= 0.1 AND playing → Pause

5. Volume:
   MediaSound → Set Volume Multiplier (currentOpen)

Gaze Detection Details

angleFactor uses direction from player to Plane, not Root’s ForwardVector
Combined trigger: openFactor = distFactor * angleFactor gives smooth gradient, not hard on/off
Ease-in curves: t*t*t (cubic) on distFactor, t*t (quadratic) on angleFactor before combining
Smooth lerp via FInterpTo prevents popping on quick head movement

Placement Workflow

Create BP_Clip instance in level
Position Root at memory’s location (desk surface, window, etc.)
Rotate Root so Plane faces natural viewing direction
Adjust PlaneSize
Assign media (set bIsVideo, fill MediaPlayer/MediaSource/MediaTexture or ClipTexture)

Media Asset Naming

One set per clip:

Asset	Naming	Type
File Media Source	`MS_Clip01`	File Media Source
Media Player	`MP_Clip01`	Media Player
Media Texture	`MT_Clip01`	Media Texture

Material: M_Clip

Shared material for photo and video. Both feed into the same BaseTexture parameter.

Properties:

Blend Mode: Translucent
Shading Model: Unlit
Two Sided: Yes

Nodes:

[TextureSampleParameter2D: "BaseTexture"] → RGB → Emissive Color
                                          → A (Alpha)
                                             → Multiply ← [ScalarParameter: "Opacity"]
                                                → Opacity

Opacity = Texture Alpha × openFactor. This single setup handles three cases:

Video: Media Texture has no alpha channel (A = 1), so opacity is purely controlled by openFactor
Regular photo: same as video, A = 1 everywhere, opacity = openFactor
Alpha-masked photo: transparent areas (A = 0) stay transparent regardless of openFactor; visible areas (A = 1) fade in/out with openFactor

Without the Alpha multiply, background-removed photos show black where the background was removed (alpha = 0 but opacity is forced by openFactor alone). Connecting the texture’s Alpha channel is required for transparent backgrounds to work.

BP_ExperienceManager

Manages overall experience arc and space decay.

BP_ExperienceManager
+-- On Begin Play: find all BP_Clip actors in level
+-- Track discovered count (listen for BP_Clip discovery events)
+-- Space Decay:
|   discoveredRatio = discoveredCount / totalClips
|   Adjust 3DGS splat count and size based on ratio
|   Lerp from full fidelity → particle abstraction
+-- Experience Arc:
    Build (timelapse of apartment setup)
    Explore (viewer moves, discovers clips)
    Decay (space dissolves as clips are found)
    Dismantle (timelapse of apartment teardown)

Sun / Sky System

Directional Light pitch rotation = sun position.

Pitch	Time
0°	Sunrise (horizon)
-90°	Noon (directly above)
-180°	Sunset (horizon)
beyond -180°	Night (below horizon)

Components:

Directional Light: rotate Pitch to change time of day
Sky_Sphere: set Directional Light Actor reference. Must call Refresh Material after every rotation change at runtime
Skylight: set to Real Time Capture for auto-updating environment light

Blueprint (Timeline):

Timeline (Float Track: "SunPitch", 0→-180 over N seconds)
  → Make Rotator (Pitch: SunPitch)
    → Dir Light → Set Actor Rotation
      → Sky_Sphere → Refresh Material

Plan: connect Timeline playback position to discoveredRatio from BP_ExperienceManager.

3DGS + Translucent Material

3DGS splats and Translucent materials conflict in rendering order. Translucent planes are invisible when placed inside a 3DGS scene because 3DGS splats render on top.

Fix: Set Translucency Sort Priority on the Plane component. Higher value = renders on top of 3DGS splats. Tried DitheredTemporalAA as an alternative blend mode, but it produces a visible dot pattern. Translucent + Sort Priority is the cleanest solution.

Integrating 2D Media into 3DGS

The 3DGS scene already has heavy texture, fragmentation, and splat artifacts at every edge. Adding rectangular photos with sharp borders looks wrong: the clean edges clash with the organic, broken quality of the 3DGS world.

What works: Alpha-masked (background-removed) photos. With transparent backgrounds, the photo content blends into the splat environment because there are no hard rectangular edges to break the illusion. The visible content (food on a plate, a view through a window) appears to belong to the space.

Design takeaway: Anything placed inside the 3DGS scene needs to be clean and borderless. The 3DGS environment provides all the texture; the inserted media should only show the subject itself.

Technical Notes

Open Source is async: calling Play immediately after Open Source fails. Enable Play on Open on the Media Player asset instead.
Instance Editable: all media variables (MediaPlayer, MediaSource, MediaTexture) must be Instance Editable, or you can’t assign them per-instance in the level.
Sky_Sphere Refresh Material: only updates in editor on manual click. At runtime, must be called via Blueprint after every Dir Light rotation change. Does NOT work in Construction Script for runtime changes.
Room scale: 4x4x4 room, TriggerDistance=500 works well.
Plane orientation: needs Y=90 rotation to stand upright.
iPhone photos: 3024x4032 (3:4 ratio), mix of portrait and landscape.

Last updated: 2026-02-17