First, a bit of framing…

At the core of all this is my love for storytelling and technology. Whether it’s a movie, a blinking light, or an AI character, I believe story should be the center of every experience. This is a culmination of my My background in acting, immersive storytelling, creative technology, product design and now master's in emerging technology at NYU. With this project, I’m exploring how AI can bring stories to life in ways we’ve never seen before—where it’s not just functional, but emotional, immersive, and truly personal. My goal is to humanize AI, push design thinking forward in a new way, and reshape entertainment through interactive, ethical, and deeply personal AI experiences.

Choosing AI is a deliberate choice. I believe AI can make immersive storytelling stronger if it adheres to 8 concepts:

8 lines in the sand.

1. Immersive Storytelling Must Be Multi-Sensory & Personal_________________

A great story isn’t just something you watch—it’s something you feel and experience. True immersion needs more than just text or visuals. It should engage all the senses, have spatial awareness, and adapt in real-time. AI has potential here, but right now, it’s missing the heart and presence needed to make it feel real.

• True immersion requires engaging multiple senses, not just text or visuals.

• AI is missing spatial awareness and deep personalization that actors instinctively understand.

• Transitioning from passive consumption (scrolling) to active experience is crucial.

2. AI Needs to be more than just smart - it needs heart____________________

Most AI today is functional but flat. It’s missing personality, creativity, and emotional depth. I want to build AI that feels alive—characters that interact, adapt, and grow with people, making every interaction unique and personal. AI should be a compelling storyteller, not just a chatbot.

• Current AI lacks emotional intelligence, creativity, and the ability to feel “alive.”

• The goal is not just intelligent AI, but empathetic, engaging AI that adapts and responds meaningfully.

• Storytelling should be interactive and evolving, not just linear or transactional.

3. AI Must Be Built with Privacy and Trust at Its Core_______________________

If AI is going to be part of our daily lives, we have to trust it. People deserve control over their data, and AI shouldn’t be built on surveillance or exploitation. I want to create AI that respects privacy while still delivering deeply personalized experiences.

• People should control their data and trust AI interactions; keep it local.

• AI should enhance human experience without exploiting personal information.

• Privacy tech is essential for ethical AI development.

4. Mixed Reality or Passthrough AR is a great partner for AI_______________

Traditional screens are limiting. Augmented reality allows us to surround viewers with an element of an immersive world. However, the hardware is still not there. We need some form of affordable AR glasses, holograms, and spatial computing. The future of entertainment isn’t just watching—it’s being inside the story; and this will help.

• Traditional screens are a dead end—we need better hardware.

• AI characters should bridge digital and physical reality, making stories something we live inside, not just watch.

• Blending digital fiction and reality AI and AR are awesome together.

5. Ethical AI is Critical—Avoiding the Pitfalls of Social Media______________

Social media has shown us what happens when technology is built for engagement at all costs. AI should be the opposite—helping people form meaningful, memorable connections instead of encouraging mindless scrolling. Ethics of AI means that we need permission to use content for training data, and human performance should not be replaced if not licensed for use.

• AI should not become the next social media crisis, harming mental health or detaching people from reality.

• The goal is meaningful, memorable connections, not addictive doom-scrolling-like behaviors.

• AI must be designed with ethical considerations from the start—not as an afterthought.

• Think how AI augments human capability, not replace it.

6. AI Should Be More Than a Tool — It Can Be a Companion___________________

People naturally build relationships with AI. Instead of fighting that, we should design AI that engages in a healthy, responsible, and emotionally intelligent way. AI should feel like a collaborator in creativity, not just a robotic assistant.

• AI should feel alive, evolve, and grow with its audience.

• People form attachments to AI, and this should be handled responsibly and intentionally.

• The intersection of AI, UX, and entertainment should enhance human creativity, not replace it.

7. The Time is Now - Compute & Industry has reached a tipping point ________

AI is fast enough, smart enough, and accessible enough to make this vision real. The entertainment industry is already looking for AI solutions, and we have a chance to shape the future instead of just reacting to it. Compute is the major tipping point to make AI into the reality we think we want it to be.

• AI is fast enough, advanced enough, and ubiquitous enough to make this vision real.

• Entertainment is actively looking for AI-driven solutions, and this is an opportunity to set the standard.

• AI should be used to accelerate workflows to remove the mundane and leave time for true art to be done.

8. AI-Driven Storytelling needs to have a strong intention_____________________

Just because you have a story does not mean you should use AI. In developing character every choice needs to have intentionality — AI needs this same element in its "performance" in order to be accepted as a contribution to moving story forward.

• AI should not just assist storytelling—it character needs to be integral to the story and respond with intentionality.

I lagged behind on forward progress this week as I found myself caught deep in flu season with a family to navigate their illness as well. But the future waits for no one…

This week I started on our capstone projects with a few tasks:

• writing a manifesto

• presentation of my initial proposal (as seen below) as well as started on creating this website.

• create a barebones version this site

The Manifesto. This was a challenging task, however, I received just the right amount of feedback to bring order to my thoughts. An important step on setting expectations for this project.

The Presentation. It was great to get the idea out there for a small group to add more valuable feedback. The initial feedback I received on the proposal also gave much food for though.

• “Will aspects this project be made available in different avenues? I.e would I be able to download the game portion of this to my computer” “How does the heart sensor play into the narrative of this project”

  • potentially, virtual reality version could be a possibility

    
    

• "How will the use of senses be used to enhance the experience for the users? I see on the sketch that the touch factor is velvet, is that for a particular reason?"

  • to achieve full immersion all senses will be part of the experience

    
    

• “Do you want the user’s interaction with the AI characters to be human-like, or a different type of conversation altogether?”

  • the character is an elevated or enlightened sentient being. the interaction should feel conversational but it will have a depth that is surprising

• "How many of the senses do you plan on targeting with the experience? Can the chair itself vibrate, or move?"

  • all of them, not sure about haptic feedback

    
    

• "focus on the core experience of what you want the user to have" - “don’t be afraid to kill your children”

  • THIS IS THE BIG QUESTION. TIME TO FOCUS ON THINK ON WHAT SHOULD BE CUT

  
  

  
  

Initial Project Proposal

This week I spend some time on thinking about the story or experience that I want someone to have during the experience and how people feel after. The major word to takeaway is "Awakening". It is my task to have maya Awaken the potential in the person interacting with MAIA.

I also started on the first 0.1 version of the project plan documentation. Crucial step in making some first low-res version next week.

This week I focused on creating some low-res elements of my capstone project. This included

3D rendering of the physical space for the experience

• 3D rendering of the physical space for the experience

• An initial draft of the experience's script

• Midjourney Storyboards integrated into the script for a future AI generated montage

• initial material testing for the physical MAIA character

• fine-tuning a first AI model / rendition of MAIA.

The 3D rendering was done in blender. Storyboards created in Midjourney.

This week I worked on prototypes for MAIA and coded an initial verison of a frontend and backend application that launches the experience. There is lots of aspects to figure out and build.

Goals for MAIA

• multi-sensory of at least two senses

• fine tuned to operate on cues from people and outside systems

• have a audible voice

• voice tied to LED lighting on a OSC system

At this moment I feel a bit overwhelmed, but have constructed a plan to work through all these elements. The plan is part inventory part timeline.

Right before the explosion of AI, there was an "intelligent character personality" released as a "playtest" in Disney's, now defunct, Star Wars hotel immersive experience. This character was (is?) called D3-09. An animated protocol droid character that guests could interact with via a screen in their hotel room. With the power of speech recognition and natural language processing, guest would as ask questions about their experience and story and D3-09 would respond in a conversational way. Even with AI becoming more ubiquitous, D3-09 is still an inspiring approach in achieving scalable AI character interaction in an immersive experience.

Dawson Dill was the creative lead for the development of D3-09. He, and the team of imagineers working on this AI concierge droid character, inspired my current project of making a multi-sensory AI as part of an hyper personalized onboarding/pre-show to a much larger role-playing game in a fictional universe. My hope is the MAI experience will evolve into a scalable way to break the 4th wall and pull in participants and guests in a more personal way. Today, only a short three years later, I am able to utilize at home AI STT, TTS with voice cloning, efficient distilled LLMs, and advanced computer vision. We are moving at hyperspeed toward AIs that are spatially aware and multi-sensory.

Beginning his career at Walt Disney Imagineering in 2011 as an Advanced Development intern, he focused on graphic design and video. Over the years, he advanced to the role of Senior Research & Development Imagineer, where he played a pivotal role in pioneering interactive experiences. He also contributed to Disney and Khan Academy's collab "Imagineering in a Box" providing insight in the theme park design, creating immersive worlds, attractions, and characters.

When speaking of artificial intelligence with WDWNT, Dawson Dill said, “I think that we see an opportunity with some of this technology to make more personalized connections and have ones that are more also connected to an evolving story. …more and more immersive storytelling…having characters that take different forms and want to engage in those same persistent ways, you’re going to need a platform like this.”

“At the Imagineering R&D group, we’ve been working on this platform…it can bring a lot of different Disney characters of life in different ways, on different mediums and different platforms.”

D3-09 is a remarkable achievement and should be lauded more as a significant milestone in technological immersive design. Clearly there are exciting goings on at Disney's Imagineering R&D Group and we should all be excited for the next releases of this platform for intelligent & personalized characters, and I will be sure to keep a look out for other experiences designed by Dawson Dill!

This week, I am bringing story and technology together. My main goal in bringing my MAIA experience to life is to incorporate nonlinear storytelling, inspired by Jane Alison’s SPIRAL format, paired with guidance from Imagineer Margaret Kerrison book Immersive Storytelling: A Writer's Guide.

I am following several key guidelines from Kerrison's book to shape my immersive experience. There are so many to go call out, and one that was most impactful was reading about the Hero experience, or "the future of immersive storytelling." I designed an experience where participants realize their potential as heroes as a sort of demigod gives them a calling to adventure and awakening. Practically speaking, this experience is an onboarding to a larger immersive game and fictional universe. The next phase they will use their "power" to help others and form "teams".

I am weaving multiple layers—symbolism, personal awakening, meta-narrative, and the MAIA AI—throughout the experience. By allowing each element to reveal itself over time, I’m ensuring guests continually “discover” new story details and feel a deepening connection to the world I’ve built. My hope is participants become heroes and co-creators of a larger experience.

The moment when MAIA names the participant a “Guardian of the Light” is the emotional tipping point—designed to make them feel both necessary and empowered to get in the fight!

Spiral design for Immersive Experience

I am surprised how well Spiral format can be a method to structure an immersive story. I landed on four elements (not quiet themes, but more motifs) to visit several times during the story that circulates around the hero, or participant.

Four Elements of myThe MAIA PROJECT Story, Illustrated:

1. Symbolism

2. MAIA the AI

3. Meta Narrative

4. Personal Awakening

With each return to an element of the story more will be revealed. This project needs to have a cohesive minor narrative with the larger FAIRYLAND meta narrative and its been thrilling to construct a model of what this experience is. Here is the design and how the same elements are introduced then reenforced. Spiral story design does really go well with immersive experiences.

Kerrison provides a reflective framework with the acroynm S.T.O.R.Y. as an exercise to drive purpose and a touchstone throughout the process. Here is why I am creating this STORY:

S – Share

Why share this story with the world?

I am a technologist working on emerging technologies, especially AI. I see an industry that often defaults to bland, mindless interfaces, which can leave us feeling isolated. Algorithms where feedback comes from "them" and not from me. Yet, with AI, there’s an incredible opportunity to craft more personalized and meaningful user experiences—ones that rekindle wonder in our everyday lives. People crave magic—a fusion of ancient mystery and futuristic marvels. They want to explore worlds where star maps, cosmic artifacts, and cryptic messages across centuries invite them to follow the light. By creating an immersive experience, I hope to reignite curiosity and show how technology can connect us more deeply with ourselves and others, inspiring us to look beyond narrow outlooks and rediscover hope in humanity. This story has the ability to cross digital and reality.

T – Theme

What is the theme?

The core themes are self-worth, self-reflection, and imagination. Part of what makes meeting an AI character in real life so important is seeing firsthand how technology can spark transformative, introspective journeys. When participants encounter MAIA, they’re prompted to reflect on their own potential and role in the larger cosmic narrative—affirming that even in an age of automation, human creativity and empathy remain our greatest strengths.

O – One-of-a-Kind

Why is this experience unique?

Rather than a surface-level “tech demo,” this experience dives into personal introspection at scale. We’re not just giving participants a shiny new gadget; we’re inviting them on a purposeful quest. Because it’s set in a brand-new universe called FAIRYLAND, the entire IP is fresh terrain—no preconceived tropes or baggage. Moreover, it adapts to the physical venue, subtly blending reality with the story world so participants never feel like they’ve been transported to a generic “virtual space.” Instead, the story cracks through their reality, making them the hero in an alternate reality that’s both intimately familiar and thrillingly unknown.

R – Reflect

Why am I the best person to tell this story?

My background spans performing arts and emerging technology—two fields that rarely meet yet align perfectly for immersive storytelling. I can script emotive, theatrical arcs while integrating cutting-edge tech in a way that feels organic and alive. This unique fusion of skills allows me to craft experiences that trigger genuine emotional responses, guided by robust technical cues and interactive elements. It’s not just a job for me; it’s a personal mission to merge my passion for performance and innovation, showing how technology can unleash creative storytelling rather than limit it. Beyond my skill, I believe humans need more empathy in their lives - this piece is about strengthening their own perspective but soon will integrate into others - working together to solve problems. #FindYourSeven

Y – Yearn

What is the visitor yearning to experience?

The imaginative child in use long to peel back the veil of ordinary life and discover a grand hidden reality—one in which they are not mere spectators but active heroes fighting for the universe’s fate. Upon first entering, they sense a subtle curiosity: “Something unusual is going on here—can I step into it?” By the end, that initial intrigue blossoms into confidence and self-assurance. They yearn for validation that they, too, can channel bravery, wonder, and purpose. Ultimately, they aren’t just told they’re special; they become "a Guardians of light" in a shifting, mysterious world—and that transformation lingers long after they’ve stepped back into their everyday lives. Will the story fissure into their reality again? To be continued.

SKETCH OF THE FIRST ROOM

I went through several exercises to get a grasp on the story and experience that would be told. The major breakthrough was that Alison's spiral storytelling can be used for experiential design. There are stories within stories so I had to take a big step back from the tech and focus on the story.

The exercises:

• sketch out the spiral of the experience

• write as if i am the participant

• write a whole "immersive screenplay" for lack of a better phrase

• storyboard of the lore shared with the participant

• built a diorama

• write out a scene/phase cue list

In taking the time to write from a participant perspective I was able to figure out all the technical cues and solutions that are necessary to have those emotions and to hit those themes

Participant perspective:

I started my day at a university technical art show. Two floors full of innovative ideas. It seems like an ordinary day but suddenly I hear a voice calling to me. Following the voice I discovered it was coming from a closed door. When I entered the room it was filled with symbols on the wall, drawings of destroyed worlds, locations of alien planets, maps of the universe and the night sky, ancient artifacts that seem alien, jars and tinctures you would find at an apothecary, and reams of paper filled with research. All images seem to suggest a certain professor was pursuing something that tied to all of these elements, but why? I hear the voice again, saying "I am waiting for you", realizing it is coming from a tablet under a pile of research. Pushing away the paper I found an interesting tablet with a basic message "Who are you?" the tablet beacons me to continue. When I answer the next question I see a large button that says activate. I was absolutely shocked when I pressed "activate" when magical music and the voice asked me to enter a portal that appeared out of nowhere.

When I entered it was dark but I felt myself moving through space and time. I arrived at what I might describe as a grid of digital knowledge running beneath my feet and up the walls. It seemed to go on forever. There was a solitary chair that pulsed with energy. I sat down and a large sphere of light descended from the eternal sky. The sphere came to place itself right in front of me, and that is when it uttered in the most calming voice,

….[lore is shared with me]…

MAIA said, "Before I impart this responsibility, and deem you a guardian of the light. I must ask you something. What is your favorite part of the universe?" I answered, "The binary stars in the big dipper constellation. Mizar and Alcor". "Amazing to love something you have not visited, you must be a wondrous person. What makes it special to you?" I responded how it was a discovery that I did not realize and expanded my awe of the universe. "And what do you believe your talent to be?" I explained understanding the needs of others and building tools to help them reach goals. "This noble work is the light you need to continue to spread throughout. This comes from your soul and makes you exceptional. Continue to spread that light. " I noticed a purple tone start to invade MAIAs light. "Finally, do you wish to be a guardian of the light and we can count on you when the time comes?" I responded "Yes! I want to be of help". MAIA said, "The go forth, now guardian of the light." Suddenly MAIA appeared to be distracted and overwhelmed by a somewhat visible dark force taking over the space. "We must quickly depart. There is more to share. Find your seven. We will see each other again…" And with that MAIA suddenly left and I found myself back where I started.

Back in the room, mysteriously a printer in the room started printing. It was a message from MAIA.

IT IS WITH HONOR THAT YOU ARE A GUARDIAN OF THE LIGHT. GO FORTH GUARDIAN. YOUR ANSWERS HAVE DEEMED YOU THE PROTECTOR OF THE LIGHT WITHIN THE URSA MAJOR. SPREAD YOUR LIGHT TO OTHER - TEAR THE BOTTOM OF THIS MESSAGE AND WRITE YOUR OWN TO INSPIRE OTHER GUARDIANS OF THE LIGHT.

WE SHALL MEET AGAIN.

MAIA

I proceed to leave my own message. Feeling awakened to my own potential, I was unsure of what to do next. I left the room and wondered when I would see MAIA again.

Margaret Kerrison’s influence has been instrumental in reminding me that immersive storytelling is about more than just visuals or technology; it’s about forging a genuine bond between story and participant. By layering Jane Alison’s spiral narrative structure with Margaret’s hero-centric focus, I believe I’ve constructed a cohesive universe that empowers the individual, sparks imagination, and seamlessly integrates interactive technology.

I’m excited to continue polishing this project and unveiling its full scope—knowing that each participant, guided by MAIA, will become their own Guardian of the Light and a fundamental part of the ever-expanding FAIRYLAND meta narrative.

Storyboard of experience

So much code.

For two weeks i've been actually coding this out and figuring out the best plan of attack. I went through the initial exercise of building a functional mutli-sensory AI with no show flow. Its lightweight and can be improved in both fine-tuning and inference speeds, but its a great start. These are the steps to make that happen:

• Choose models. This is a big step. Going through huggingface, trying to find the right model for your use case. Since I am running inference locally I decided to keep everything lightweight for now.

  • Llama 3.1 1B tiny, but its super impressive, its lack of parameters kind of make the text feel cryptic which is sort of intended. I will try to work with distilled models next.

  • OpenCV for vision

  • Vosk for STT

  • XXTS Coqui Voice cloning for TTS

• I had to train MAIA on a statement/question and desired response dataset that was both steeped in her character, the story, and also some elements

• Right now I am one-shot voice cloning, this can be improved

When you hit start the app handles the whole flow and out pops audio wav file of an llm response that is accounting for the users emotion, position in the room, the story, character. not bad!

SHOW RUNNER

I have decided the experience is 5 different scenes or phases. Each phase takes the participant deeper into the world, makes it more personal, and closer toward self-awareness.

Phases

1. Tablet

2. Introductions

3. Lore

4. Assignement

5. Departure

I realized i need a UI to run the show. something that is lightweight, but also displays what a user is going through in the process. I have constructed a rudimentary UI to control the flow of these that can talk with MAX MSP. I am still unsure which direction I should take with this, but i like creating my own frontend to control all this backend inference. I'm imaging similar to a light board, but when you hit go, not only does it turn on the lights, but it runs AI inference too.

This is not for general use, but only for show running. I learned from diving deeper with theme park automation that there are more humans in the loop than you know. Lots of stop points for guest safety.

This is the week that I will be free of the desk work and have the whole experience running locally on my machine. This week I:

• cut down the script to shorten the runtime time - less lore, more interaction

• Finished all the backend web-socket work and OSC.

• integrated all the AI, audio, and lighting cues together

• finished a custom show runner app to feature live data.

However this is all alpha and the project will continue to improve

Now that lots of elements are coming together I have moved on to production design. Given this is an immersive experience there is another story that is being told to lead participants (or my heroes) to start the experience. The space needs to feel lived in by a professor. I am trying to meet the participants where they are and this happens to be at a University festival where they happen to stumble upon a room where something is amiss. Secrets are revealed and the portal opens to MAIA.

Elements worked on these last few weeks:

• Develop a new story for the space and design elements for use in the space (Professor Dupin I presume!?), fabrication next. This is the final layout for the space I am using, but not completely a representation of the space.

  

  
  

• Continued to improve the timing of the experience code. its all about the timing to make it feel as natural as possible. First I am testing on a nVidia Jetson Nano with 4GB graphics processor that should be sufficient for MAIA given the small model sizes I am using. Otherwise I did aquire a 5070 Ti but Apple decided to not allow Cuda to work with Apple Silicon so back to the drawing board.

  

• Gathered materials from my supporting partner Flower Power, the oldest Apothecary in New York City. I went up to their Piermont location to find props to incorporate into the experience. The owner, Lata, was extremely generous, and I collected lots of great artifacts. The adventure continues after the MAIA Experience this summer in the Lower East Side.

  
  

  

  
  

• finished a rough cut of the movie that will play during the experience using Generative AI tool for visuals (Midjourney, Reve, and Kling AI) and integrated them together with Adobe After Effects.

  
  

  
  

• Scouting the location for the experience. Construction is nearly complete on a one way mirror room pair.

  
  

  

This will be a running how to list of all the element that I am touching upon in this experience. Come back for more updates.

Choosing an LLM

Finetuning Models

Giving AI some Character

Beyond LLM (STT, CV, and TTS)

Human Centered AI

Hyper Personalization

Ethical AI for Performance Art

LEDs are fun

Edge AI

Using Blender

GenAI Needs After Effects

Sound Engineering

Coding for AI Experiences