Give Your AI Characters Eyes & Ears in Unreal Engine: Streaming Vision + Hands-Free AI Voice Interaction

Convai’s Unreal Engine plugin enables real-time voice AI with streaming vision, letting your AI characters see what you see and converse hands-free. According to MarketsandMarkets, the voice AI market is expected to grow at a CAGR of 17.2%, highlighting the rising demand for such immersive interactions.

Now imagine you’re on Mars. The solar network is down, comms are failing, and an AI voice in your suit says:

“Your mission objectives are to repair the solar power network. Can you do a quick check on the main panel?”

You look up at the array and ask: "Hey, what is that?"

The AI answers based on what it sees from your helmet camera, and you never touch a key.

This is what Convai’s Unreal Engine plugin (Beta) is built for: streaming vision input + hands-free voice. It lets you build an AI character that can see the world and talk you through it in real time.

In this guide, we’ll walk through exactly how to build that experience:

1. The Setup: Installing the plugin and wiring the AI into your player.
2. The Voice: Flipping from "push-to-talk" to hands-free.
3. The Vision: Giving the AI a live camera feed so it can see what you see.

See it in action in the detailed walkthrough below:

What this plugin actually gives you

Conceptually, you’re dropping a Jarvis-style assistant inside your character. It lives in the same blueprint, sees what the player sees, and talks them through missions.

Powered by the Live Character API (WebRTC), you get:

• Low-latency conversation: It feels like a live teammate, not a laggy chatbot.
• Hands-free speech detection: No constant key-holding required.
• Streaming vision: The AI processes a live video feed from an in-scene camera.

How do I install the Convai Unreal Plugin?

You only have to do this once.

1. Download: Go to the Convai Unreal SDK releases page on GitHub and grab the latest .zip.
2. Install: Extract the Convai folder.
   
   • Engine-level (Recommended): Drop it into Engine/Plugins/Marketplace.
   • Project-level: Drop it into YourProject/Plugins/Convai.
3. Enable: Open your project, go to Edit → Plugins, search for "Convai", enable it, and restart the editor.

How do I connect my Convai account in Unreal Engine?

Once Unreal restarts, the Convai setup panel should pop up.

1. Log in with your email or Google account.
2. Once authenticated, you’ll see the Convai dashboard inside the editor.

Your engine is now connected. Time to wire a character into your player.

How do I add AI “Ears” and “Brain” components to my player?

Technically, we are adding Convai components directly to the Player Blueprint.

How do I find my Player Blueprint?

If you aren't sure which blueprint controls your player:

• Press Play.
• Click Eject in the viewport to detach the camera.
• Select your player character in the World Outliner.
• Click Edit Blueprint in the Details panel.

How do I add the "Ears" (Convai Player Component)?

In the Components panel, click + Add and select BP_ConvaiPlayerComponent. This handles microphone input and the chat UI.

• Test it: You can hit Play right now. You should see Convai’s on-screen widget (push-to-talk is on by default).

How do I add the "Brain" (Chatbot Component)?

Click + Add again and choose BP_ConvaiChatbotComponent. This handles the reasoning and API connections.

Now, give it a personality:

1. Open Convai Playground in your browser.
2. Create or select a character (e.g., "Mars Mission Assistant").
3. Copy the Character ID.
4. Back in Unreal, select BP_ConvaiChatbotComponent and paste the ID into the Character ID field.

Compile & Save.

Hit Play, hold T (default), and say "Can you hear me?" If it responds, "Acknowledged, astronaut," you've got a voice in the suit.

How do I enable hands-free voice interaction?

Push-to-talk is great for testing, but for a "working" simulation, you want flow.

1. Select BP_ConvaiPlayerComponent.
2. In the Details panel, uncheck Enable Push To Talk.
3. Compile and Play.

Now just talk. "Hey, are you online?" The AI should reply: "Acknowledged. I’m listening."

How do I give my AI character streaming vision?

Now for the fun part: giving the AI actual sight.

How do I add the webcam component for AI vision?

1. In your Player Blueprint, add an EnvironmentWebcam component.
2. Important: Drag the EnvironmentWebcam onto your main camera (e.g., FirstPersonCamera) to parent it. This ensures the AI looks where you look.
3. Set its Location and Rotation to 0,0,0 so it aligns perfectly.

How do I create a Vision Render Target?

1. In the Content Browser, right-click and choose Convai → Vision Render Target. Name it RT_Vision.
2. Go back to your Player Blueprint → select EnvironmentWebcam.
3. Set the Convai Render Target to your new RT_Vision file.
4. Make sure Auto Start Vision is checked.

How do I test streaming vision with my AI?

Hit Play. Look at something in your scene (like a solar panel or a specific tool) and ask:

"Hey, what is that?"

You might get a response like:

"I see a solar array. The main dish is angled toward the horizon and there appears to be dust buildup on the lower struts."

Follow up naturally:

"Is it aligned correctly?"

"Negative. Initiating alignment sequence for optimal signal lock."

That’s streaming vision + conversation. The AI isn’t guessing; it’s describing exactly what your camera sees.

What are some use cases for streaming vision and hands-free voice AI?

Once you’ve got this pattern working, you can reuse it in almost any Unreal project:

• Field Training: Trainees point at a machine and ask, “Is this set up correctly?”
• Maintenance Coach: An AI that validates you are looking at the right fuse before you pull it.
• Guided Tours: A virtual museum guide that explains whatever artifact the player approaches.
• Accessibility: An assistant that describes the environment to visually impaired players.

What troubleshooting tips can help if something isn’t working?

• No AI Response: Double-check your Character ID in BP_ConvaiChatbotComponent.
• Hands-free not working: Ensure "Enable Push To Talk" is unchecked in BP_ConvaiPlayerComponent.
• Vision not working: Verify EnvironmentWebcam is parented to your camera and Auto Start Vision is checked. Open the RT_Vision file while playing to see if it's capturing video.

(Remember: For support, feedback, or to report bugs on the Beta, jump into the Convai Developer Forum.)

FAQ

Do I need C++?
No, not if you install at the engine level. Project-level installs do require a C++ project, but the engine-level Marketplace route works fine for Blueprint-driven games. 

Can I use this for characters that aren’t “in-suit”?
Yes. In this tutorial, the AI lives in the player blueprint (like an in-helmet assistant). For classic AI characters placed in the world, you’d attach BP_ConvaiChatbotComponent to that character’s blueprint instead. 

Where do I get updates and docs?
The latest plugin releases and setup guides are available in the Convai Docs under Unreal Engine Plugin (Beta) → Installation & Setup. ‍

What is Convai’s Unreal Engine plugin?‍

Cnvai’s Unreal Engine plugin integrates streaming vision and hands-free voice AI into your game characters. It allows NPCs to perceive the environment through live camera feeds and interact naturally with players using real-time voice recognition.‍

How do I make an NPC in Unreal Engine respond to the player’s voice in real time?

Using Convai's SDK, you add components to your player or NPC blueprint that handle microphone input and AI responses. The plugin supports hands-free speech detection and low-latency conversation, enabling NPCs to reply instantly to voice commands.

‍Can Convai’s plugin work without coding in C++?

Yes. If installed at the engine level, Convai’s plugin supports Blueprint-driven games without C++ coding. Project-level installs require a C++ project, but the engine-level Marketplace route is easier for non-programmers.‍

Does Convai support lip-sync for AI characters?

Convai’s NeuroSync technology enables real-time lip-sync for more realistic NPC interactions.

‍Where can I find documentation and updates for the Convai Unreal plugin?

You can find the latest plugin releases, setup guides, and documentation on the Convai Docs site under Unreal Engine Plugin (Beta) → Installation & Setup. For support, visit the Convai Developer Forum.

‍What does Convai’s Unreal Engine plugin actually provide?

Conceptually, you’re dropping a Jarvis-style assistant inside your character. It lives in the same blueprint, sees what the player sees, and talks them through missions.