On February 7, 2023
What happens when XR developers “have a chat” with ChatGPT?
ChatGPT can have a significant impact on XR development by helping developers to work more efficiently and effectively. ChatGPT can assist XR developers in so many ways, you should go at it! Why?
- Faster guidance: ChatGPT can provide and summarize relevant information about XR SDKs, frameworks, libraries, functions, etc., and suggest how to use them in XR projects.
- Faster prototyping: ChatGPT can generate code for XR applications, saving developers time and effort when creating basic script templates, code for specific XR interactions, import of XR assets, inspection tools in the Editor, tests, etc.
- Higher productivity: ChatGPT can assist in generating, creating, and importing content for XR projects, such as text, images, and 3d assets.
- Improved code quality: ChatGPT can generate well-commented code, providing an overview of how it works. This can help XR developer teams to understand the code better and make necessary adjustments.
- Higher efficiency: ChatGPT can assist in debugging, optimizing, testing, documenting code, and other highly repetitive tasks, saving developers time and effort and allowing them to focus on more complex tasks and to iterate faster.
This article looks at how ChatGPT can make Virtual Reality and Augmented Reality development easier, more efficient, and faster:
- by asking ChatGPT for guidance and relevant information about specific XR libraries, frameworks, single functions
- by generating code with ChatGPT
- by creating auditory and visual content for Unity XR applications with ChatGPT
- by optimizing and debugging the code of XR apps and games with ChatGPT
- by writing documentation for XR apps and games with ChatGPT
Ultimately, I will add my view on whether ChatGPT will change my daily life as an XR freelancer.
What is ChatGPT?
ChatGPT is a cutting-edge AI tool created by openAI. It was released in November 2022 and gained 1 million users in just five days. GPT-3 can understand the context of the conversation, which is a significant advancement in chatbot technology. We can take advantage of this ability and “teach” ChatGPT to get our desired results.
ChatGPT can assist XR developers in several ways to produce XR prototypes and applications faster.
1. Use ChatGPT as a Pool of Valuable Information
ChatGPT potentially enhances the way XR developers search for and access information. While it only partially replaces traditional search tools like Google, Stack Overflow, etc., ChatGPT will become a valuable search tool.
Without further ado, let us take a look at the following examples:
This is already a good starting point. However, whereas ChatGPT is right about OpenCV, we wish it was already possible to access the camera feed to run computer vision algorithms in an XR project that targets the Oculus Quest. ChatGPT’s last suggestions (Leap Motion, OpenHandModel) look weird, especially the latter one XD it started “hallucination.”
So, let us ask a more specific question:
Instead of using OpenAI’s UI for ChatGPT, we can even speed up such research by using a web browser extension and sending such requests while googling. As you may see, we can get code snippets, explanations, and a summary of how to debug and fix issues along the search engine results.
2. Use ChatGPT for Text-Based Scripting (creation and finding mistakes)
ChatGPT can generate scripts for VR environments and interactions, allowing developers to prototype and test ideas quickly.
For the following example, we fed in the below prompt to generate a C# script for a Unity3d and XR Interaction Toolkit project.
ChatGPT generated the below output:
First, an abstract.
Followed by the code. In this case, an entire C# class is ready to use in Unity3d. It generated even two lines to unregister
Listenersand a line to do a
public class CreateAndChangeColor : MonoBehaviour
public GameObject myCubePrefab;
private GameObject myCube;
public XRController leftController;
public XRController rightController;
public float scale = 0.5f;
public Color startColor = Color.red;
public Color endColor = Color.blue;
private void OnEnable()
private void OnDisable()
public void InstantiateCube(XRBaseInteractor interactor)
myCube = Instantiate(myCubePrefab, leftController.transform.position, Quaternion.identity);
myCube.transform.localScale = new Vector3(scale, scale, scale);
myCube.name = "myCube";
myCube.GetComponent<Renderer>().material.color = startColor;
public void ChangeColor(XRBaseInteractor interactor)
if(myCube != null)
myCube.GetComponent<Renderer>().material.color = endColor;
Source: ChatGPT v3
And finally, it also output a short explanation and “How-To” guide to let us know what should be done in the Unity Editor:
Of course, we should always use our judgment to make sure that the code and the instructions are correct. Thus, take the result as a starting point and maybe a time-saver.
We gave ChatGPT a hint and the following task to correct a few lines of code:
The relevant code snippet modified by ChatGPT looks as follows.
private void OnEnable()
private void OnDisable()
Source: ChatGPT v3
Another use case of text-based scripting with ChatGPT is the generation of Unity Editor scripts. For example, a small tool to count the polygons of a mesh is useful while optimizing an XR app.
Neat. However, ChatGPT-3 is known for its ability to generate persuasive yet incorrect answers. Stack Overflow banned users from posting replies or comments produced by ChatGPT. (AI-generated answers temporarily banned on coding Q&A site Stack Overflow )
The moderators of the platform explained:
The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good.
Sample: On-the-fly code generation …
Now, let’s take a look at a more advanced use case, a prototype by Dilmer Valecillos. His prototype demonstrates that we can utilize ChatGPT to generate on the fly code for use in Unity3D.
After setting up a Python environment including ChatGPT Wrapper and dependencies, we create a ChatGPT Web Service (API) in Python. This wrapper will run a headless browser using Flask. ChatGPT Wrapper
To test the communication between ChatGPT and the custom web service, we set up a VS Code REST Client. The service sends prompts to ChatGPT and, in return, receives code from ChatGPT that is generated based on the criteria included in the prompts. All data from and to Unity is sent as JSON data.
The prototype calls the custom service from within Unity to communicate with ChatGPT.
After parsing this JSON response, the generated code is then run in Unity at runtime. For example, we let ChatGPT generate a
Monobehavior script to instantiate a sphere and rotate it over time which is then added to the scene and executed. With Dilmer’s prototype, we even let ChatGPT generate a script to import a car asset randomly picked from Sketchfab, to instantiate and animate it.
A prompt is sent to ChatGPT, at runtime. The generated, returned code is attached as a MonoBehavior to the scene. Then, an asset is downloaded from Sketchfab. Source: Dilmer Valecillos
The downloaded and imported asset (a car) is in action. The car Transform is being translated by the generated code. Source: Dilmer Valecillos
The prompts for ChatGPT are created at runtime in a separate component. They specify the task, the name of the generated script (class), the method, the parameter, and entity names, as well as animation parameters. A set of custom reminders and rules is appended to each prompt to ensure that the generated code is formatted correctly, includes required namespaces, and uses the correct types.
The prototype also includes a feature that allows the user to retry the code generation process up to three times in case of errors. This feature is useful for ensuring that the generated code is functional and can help to mitigate errors that may occur during the code generation process.
In conclusion, the prototype demonstrates the potential of using ChatGPT for on-the-fly code generation in Unity3D. It allows for the automation of repetitive tasks and can be used to speed up the game development process. The use of ChatGPT in this way opens up new possibilities for automating and streamlining the process of creating Unity scripts.
3. Use ChatGPT to Generate Dialogues and Voice-over Scripts
To enhance the user experience, ChatGPT can generate dialogue and voice-over scripts for XR characters and language learning.
Incorporating a text-to-speech system, which converts the chatbot’s written responses into speech, allows the app user to listen to the chatbot’s output text rather than read it. The advantages of using text-to-speech with ChatGPT are clear:
Firstly, it makes the chatbot more accessible to users who prefer auditory information overwritten. This can be particularly helpful for users with visual impairments who may have difficulty reading text on a screen. Additionally, text-to-speech can make the chatbot’s responses sound more natural and human-like. The conversion allows for the inclusion of appropriate inflections and emphasis, making the conversation feel more authentic.
We used the Oculus Voice SDK for the following test to convert the user’s speech to text. This text is fed into ChatGPT as a prompt. Once ChatGPT returns its output as JSON data, we parse it, convert it to an audio file and play the latter. Let us watch it in action:
Sample audio files generated on the fly in Unity with ChatGPT and Text-to-Speech:
- Question: Describe the dog breed, Saint Bernhard.
- Question: Who is Leonardo da Vinci?
- Question: What is Arachnophobia?
- Question: Describe the dog breed Newfoundland.
For a more advanced use case, check out the video by Sercan Altundas (Ready Player Me) who creates smart NPCs (non-player characters) using Ready Player Me avatars, Unity Starter Assets, and the OpenAI API
4. Use ChatGPT to Create Visual Content
ChatGPT can generate text and visual descriptions for XR environments, objects, and characters, which can be used in mood boards as reference material for artists and designers.
However, creating art prompts can be challenging, but with the help of ChatGPT, it can become much more manageable. ChatGPT is a powerful language model that can generate creative and detailed text descriptions, a crucial aspect of writing art prompts.
One effective strategy when writing art prompts is to divide them into four groupings:
- Content type
By breaking it down this way, it becomes easier to focus on specific aspects of the prompt and add more detail.
For example, adding more detail would lead to more exciting and creative results when it comes to the description. We can ask ChatGPT to generate more detailed text descriptions to achieve this.
Once we have a detailed text description, we can feed it into another AI that generates images, such as DALL-E (also by Openai), Midjourney, and Stable Diffusion. With this combination, we can generate various visual styles and compositions, making developing new and exciting art prompts easy.
Another advantage of using ChatGPT is that we can continue asking for more information, such as different lighting, illustration styles, or composition ideas. With its capabilities, ChatGPT can help us create art prompts that later result in detailed and visually exciting imagery generated by another AI.
Results of a quick test with ChatGPT and Midjourney
To start, we fed our prompt to Midjourney:
image is a panoramic, spherical, 360° view taken from the top of the tower “Alter Turm”, in xxx Germany. It is a sunny day, blue sky view on the streets of xxx. photorealistic style. spherical, 360°
We asked ChatGPT for more detail, it outputs the following, which we then input to Midjourney:
ChatGPT is a powerful tool that can assist in creating art prompts, particularly for XR developers and XR designers. Its ability to understand and interpret human language allows for the generation of detailed art prompts to generate visually interesting imagery through other AI systems.
Quick Intersection: How to write prompts effectively
- Define the topic Start by defining the topic of your conversation with ChatGPT, this will help the model understand the purpose of the conversation and provide more relevant answers.
- Define a task clearly Clearly define a task for the model to complete. This will give it a clear direction and help it understand what you want it to do.
- Ask specific questions Ask specific questions within your prompt to give the model a better understanding of what you’re looking for.
- Let ChatGPT ask questions Tell ChatGPT it shall ask questions before returning any output to understand the task better.
- Evaluate the output After receiving a response, evaluate it. If it’s different from what you were expecting or needs more detail, it is necessary to refine the prompt. Make sure your prompt is specific and concise. Include the questions you want to ask and the context of the conversation, and consider the tone of the conversation as well.
- Repeat Use the refined prompt to test the output from ChatGPT. Ask questions and provide examples of the desired output.
- Do not stop thinking on your own! Always validate and evaluate the output.
5. How to test and debug your VR/AR App with ChatGPT
ChatGPT could assist in testing and debugging VR applications by identifying potential issues. It has promising potential and might set a standard for (low-code) testing solutions to many issues in the future (with emphasis on “future”).
For a simple test, we fed our basic OpenAIClient class into ChatGPT, together with the following prompt:
(NB: This is a very basic class, do not use it in production)
public class OpenAIClient : MonoBehaviour
private const string URL = "https://api.openai.com/v1/completions";
private string _modelName = "text-davinci-003";
[ ] private InputField _inputPrompt;
[ ] private InputField _inputResults;
[ ] private GameObject _loadingIcon;
[ ] private GC_TextToSpeech_Chatgpt _textToSpeech;
private string _apiKey = null;
private bool _isRunning = false;
public void Execute()
_isRunning = true;
if(!string.IsNullOrEmpty(_inputPrompt.text) && _inputPrompt.text.Length > 10)
private void Start()
private void MakeRequest()
RequestData requestData = CreateRequestData();
string jsonData = JsonUtility.ToJson(requestData);
byte postData = System.Text.Encoding.UTF8.GetBytes(jsonData);
UnityWebRequest request = UnityWebRequest.Post(URL, jsonData);
request.uploadHandler = new UploadHandlerRaw(postData);
request.downloadHandler = new DownloadHandlerBuffer();
request.SetRequestHeader("Authorization", "Bearer " + _apiKey);
For example, we expect ChatGPT to tell us that
- we do not dispose of the
- we shall not store the API key in the directory
UnityEngine.JsonUtilityis not the best library to parse JSON data
Here is the output:
- ChatGPT detects that the script uses a
UnityWebRequestwhich is correct, but it did not recognize that we call
requestobject that returns a
UnityWebRequestAsyncOperation asyncOp = request.SendWebRequest();
- ChatGPT correctly detects that the script loads the API key from the
- Well, we actually check whether the
request.erroris null or an empty string, when the
UnityWebRequestAsyncOperationhas completed. But, we could improve the error handling
- Not disposing of the
UnityWebRequestis detected by ChatGPT.
- ChatGPT tells that JsonUtility is not the best library for JSON data handling
Though not all of the suggestions by ChatGPT are correct, they may give valuable hints and point out how the code could be improved.
Regarding Unit Testing, we will skip this for now because it does not make much sense to test this on the above OpenAIClient.cs. Unit tests cannot be applied to asynchronous methods. But, I want to point out an article about a Visual Studio Extension that links ChatGPT to this IDE to generate Unit tests.
6. How to Write Virtual Reality Game and App Development Documentation with ChatGPT
Writing code documentation is tedious and time-consuming for developers, particularly for large and complex projects. One of the main challenges that product teams face is keeping the documentation up-to-date, consistent, and easy to understand, which requires a lot of time and resources. But what if they could use a tool that could assist in creating and maintaining their documentation? That’s where ChatGPT might come in.
One of the ways ChatGPT can assist developers is by providing instant access to relevant information and examples. For example, when a developer team writes documentation for a particular function, they could query ChatGPT for details on how to use that function or method. ChatGPT would provide the appropriate syntax and examples. This saves time and effort, as developers would not have to search through multiple sources or rely on their memory to recall the correct syntax and usage.
ChatGPT could also be helpful by providing suggestions for common pitfalls and mistakes to ensure that the documentation remains comprehensive and easy to understand. For example, ChatGPT could provide suggestions for handling common cases, such as invalid input or missing arguments, if a method takes an argument.
ChatGPT could also be used to verify the accuracy of existing documentation. For example, if developers are unsure about the syntax or usage of a particular function, they could ask ChatGPT for confirmation and clarification.
In complex projects, multiple contributors are working on the creation of documentation. It is challenging to keep a consistent style. ChatGPT could check for spelling and grammar errors and suggest templates and boilerplate text. This helps ensure that the documentation is high quality and easier to understand.
Let’s do a small test and let ChatGPT generate documentation of our above
We fed in the following prompt:
And without further ado, here is the generated output:
This looks promising for more complex projects since the generated descriptions and comments are correct. The output has already formatted text and it provides suggestions for common pitfalls and mistakes.
ChatGPT is going to change our daily life as an XR developer.
As a freelance XR developer, I am responsible for implementing XR projects and the ideation, deployment and roll-out, and communication with clients and stakeholders. Just as written in the six steps above, I have utilized ChatGPT as starting point throughout the entire XR development process, from ideation to deployment: I rely on ChatGPT to generate ideas and inspiration for the game and level design, as well as for creating content for team communication with designers and presentations for clients and stakeholders. Additionally, I utilize its capabilities to troubleshoot code and Git issues, generate code snippets, optimize and document code, and even create and modify content within Unity XR projects in real-time. Furthermore, I let ChatGPT generate the necessary documentation and required content for the roll-out of XR applications.
We must remember that ChatGPT v3 is trained on data until 2021. It may need to be fully aware of the more recent updates and advancements in SDKs, frameworks, and APIs. However, we can continuously update its knowledge by providing it with the latest information to teach it accordingly.
Summa summarum, ChatGPT, and generative AI has significantly enhanced and accelerated our entire XR workflow.
What might the future of XR development look like?
As ChatGPT continues to evolve and improve, it will become increasingly capable of generating more human-like responses and understanding the context of conversations. With a more powerful and efficient GPU, the model can process data faster and produce more accurate results. These advancements will benefit XR developers as they continue to create immersive experiences.
In the future, ChatGPT and other generative AI will become essential tools for scaling the creative process and iterating on ideas. While coding will remain essential and generative AI tools will only partially substitute for human input, expertise, and judgment, combining multiple generative AI outputs will replace much of the repetitive work that needs to be considered creative. This includes tasks such as testing, deploying, and writing documentation.
To be a successful XR developer in the future, having a potent combination of deep domain expertise and an action-oriented prototyping mindset will be more than ever crucial. Understanding various generative tools and how to incorporate them into XR applications effectively is essential, from the initial stages of ideation to deployment. This includes having a holistic understanding of the relationships and interfaces between different components and XR-enabling technologies, such as 3D engines, micro-services, serverless backends, visual scripting, digital twins, and AI assistants, and how they can work together seamlessly within the larger XR ecosystem.
As a final tip, we would recommend using ChatGPT wherever you can starting from now (it should be easy with all our tips above). You will gain a competitive edge over others and will even be able to offer “generative VR Dev” services. It always sounds crazy that some developers do not want to use generative AI services, many developers out there are not using generative AI to support their development services yet. That’s why the earlier you can get your foot in, the more you can distinguish yourself from others.