SUBJECT SPECIALIST AREA

This is where I will post all of my work regarding the specialist subject.

Proposal

04/11/16

One aspect of video game making that is overlooked is Audio. For my specialist subject I have chosen to find out how audio is recorded and then implemented into games through Unreal Engine 4. I will be looking into the roots of how audio is recorded from Foley artists in the studio to the Audio Engineers and then how it is brought into the game for surreal sound effect. The reason as to why I chose audio is because I want to use this opportunity to focus on something which scares me and usually audio is rarely covered in games so I wanted to explore this field more. I will be looking at how audio is used and reverbs, so how vibrations affect audio in different places within an 3D Engine.

My inspiration for this subject came to me when I was thinking of my strengths, I listed down things which I am good at which includes Unreal specifically Blueprints. That got me thinking what skills am I lacking in or I could improve that could also help me with another unit, essentially killing two birds with one stone. So I came across audio and I felt like I can make this work because it’s something which important to every game, film and any media product. So in a sense my inspiration came from my curiosity, I looked for something that scared me and I ended up going with something that fascinated me instead so I am going to follow this through.

The thing about audio is that you can’t improve it, you have to make it sound more clear so that’s possible through two different ways. One is by playing around with audio settings until it’s as clear as it can be or to use better sound equipment, I will be exploring both options and might even use both methods.

Current trends in audio is that reverbs are being used more efficiently because you now have sounds which react to different textures, materials and environments. This is major trend which used in most games now and the reason is that it gives a sense of immersion to the game and adds to that little detail.

My question is ‘How is audio used to create an immersive experience’. So I will be talking different recording methods and how they make sounds, I will cover both Foley artists and artificial sounds. These two are distinctively different because a Foley artist makes the sounds from scratch e.g. they will use celery to get the sound of a bone breaking. Artificial sounds are different because they are made by using computer software’s such as FL Studio to create unique sounds. By the end of the task I want to be able tell anyone how sound works in games and to be able to understand the structure behind immersive sounds within games. For feedback I will use Questionnaires because from past experiences, I feel that they’re the best method in order to get critical feedback.

Games to research:

RAIN

INSIDE

Batman: Arkham Knight

Research videos regarding sound:

https://www.youtube.com/watch?v=g5IMQ12hI3w

https://www.youtube.com/watch?v=CfwDLONo7nI

https://www.youtube.com/watch?v=lR3zt_Wzykw

Time Chart

04/11/16

Research and Planning

11/11/16 – 18/11/16

I am going to be looking into different ways to record and edit sounds in order for them to be used in a games engine. So I will cover Foley artists for natural sounds and then different programs which mix different natural sounds to make an unnatural sound.  Next I will be looking into different game engines and which will be the best for my project, at the moment I have two main focuses which are Unreal Engine 4 and RealSpace3D.

Recording Sounds

http://www.premiumbeat.com/blog/recording-foley-and-sound-effects-the-fundamentals/

This was the first website I came across and instantly I found out that Foley artists are used only for convincingly depicting visuals which require it such as a bone breaking, anywhere else they use stock sounds. The website also gives tips on how to set up a soundstage if you’re thinking of recording sounds and it says that a large interior would be more preferable because ‘The distortion of the depth, width, and height can be minimized by incorporating sound diffusion and sound absorption to nullify the room boundaries.’ This basically means that I larger room can help the recording device focus just on the sounds you produce. It goes into depth on how to produce a Quadratic Diffuser which is essentially ‘an effective tool in the management of standing waves in a recording or mixing environment’ (https://www.youtube.com/watch?v=ja__O7JXpPw) This video gave me a very detailed way to make one if I ever need to.

This is what a Quadratic Diffuser looks, these would usually cover most walls in a professional recording studio.

Next they explained how Foley artists decide what objects to use when recording their sounds. They start off by reviewing the movie or game and when they have a sense what sounds are needed; they compile a list of sound effects in order. Afterwards they get to work in deciding what tools they need to use in order to make the sounds; there are a few methods which Hollywood Foley artists have used according to the website:

• Thin sticks and dowel rods produce excellent whooshing SFX
• Old chairs and stools are perfect for controlled creaking
• Heavy-duty stapleguns serve for excellent gun noises
• Roll up a large phonebook for realistic body punches
• Twist and snap sticks of celery for convincing bone breaks
• Corn starch inside a leather pouch makes the sound of snow crunching
• Cut a coconut in half and line them with a soft material for a horse walking
• Ball up and walk over old audio tape for the sound of grass footsteps
• Locate an old car door or fender to produce metal and car crash sounds
• Flap a pair of cleaning gloves for the sound of realistic bird wings

They use these methods and then mix and match them with other techniques they know or have thought of themselves in order produce the right sound effect. https://youtu.be/GrbgY6ajTgo This video features an Emmy nominated Foley artist and she demonstrates the basic methods such as using celery in order to get the sound of a bone breaking and cannoli pasta tube to get the sound of an internal bones breaking. The smart thing is that they sometimes put the cannoli in their mouths in order to get a different sounds which mirrors the sound of a nose breaking really well and so subtle differences like that can be very impacting.

The article stated that microphone selection and placement is very important. Shots that are up-close might need a different placement to shots which are distant. The execution is mostly improvised because Foley artists depend on basic equipment making unique sounds so nothing can be planned to in depth beforehand.

Software’s

I have to look at the different software’s which I can use for my final project so I will go over my options and then decide which one is the best based on that.

Unreal Engine 4 is suite of integrated tools for game developers to design and build games, simulations and visualisations. Main is to obviously make games and quite a number have been using this including Dishonoured, Batman: Arkham Knight and more recently Gears of War 4. If you have seen the sheer scope of these games then you have an idea of what UE4 is capable of doing, but for this project I will be looking at the audio aspect of it.

First I have to decide what engine I will use in order to bring my project to life, I have one main focus and a smaller engine which serves as a sort of plugin. My main choice would be Unreal Engine 4 because in my experience it has been the most powerful and easy to use so far, but also I want to expand my skillset within it so doing this will further increase that skillset.

https://docs.unrealengine.com/latest/INT/Engine/Audio/

This webpage is the Unreal Documentation on Audio within the engine, here it can tell me everything the engine has to offer in terms of sound and audio tools. It covers the importing of audio and sounds which is really basic stuff. Next it delves into sound cues which talks about using blueprints and nodes in order to mix and loop sounds together, so it’s basically an inbuilt sound editor. There are a lot of different Nodes that you can use to get unique sounds.

Sound Nodes

This is how the different nodes look like when placed into a blueprint, I have also listed all of the nodes below and highlighted the ones that I will be looking into.

• Attenuation Node
• Branch Node
• Concatenator Node
• Continuous Modulator Node
• Crossfade by Distance Node
• Crossfade by Param Node
• Delay Node
• DialoguePlayer Node
• Doppler Node
• Enveloper Node
• Group Control Node
• Looping Node
• Mature Node
• Mixer Node
• Modulator Node
• Oscillator Node
• Random Node
• SoundClass Node
• Switch Node
• Wave Param Node
• Wave Player Node

All of these nodes work as in-engine sound editors, you can set the frequency, pitch, loop, mix the sounds etc. The ones I highlighted caught my attention because they seemed like they could help me set the audio and sounds cues for my scene. I won’t be discussing all of them because I might not end up using all of the ones I highlighted, but I will talk about how the nodes might not do what you think they do. For example, the Loop Node doesn’t loop the sound, you use the Wave Param/Player Node for that so I will be reading through the documentation before I use any of them.

Next it goes through a detailed of the Ambient Sound Actor which can be used for both looping and non-looping sounds, it basically makes the sound louder the closer you are to it so this is defiantly something I will be looking into considering there’s a lot of modifications for it.

Next it goes through something really interesting, Sound Attenuation. It basically makes smart sounds in a sense, it gives sound the ability to lower the volume as the player moves further away from it. This a really smart feature and can be used in blueprints which will blend nicely with the other nodes so this is one to keep an eye out for.

When I was researching for the engine I found something really interesting, an audio engine which can also be used as a plugin for Unreal. It’s called RealSpace3D Audio and it ‘allows Game Designers to create extremely immersive audio that enhances their 3D visual offerings’. (Source: http://realspace3daudio.com/technology/) The plugin itself is really complex because it creates so immersive and complex that it borders the uncanny valley in the sense that you won’t be able differentiate the sounds from a game to real life. That’s the kind of stuff this plugin specialises in so it will be interesting exploring the different features it has and how it will blend in with Unreal’s sound features.

Recording Sounds

27/12/2016

So today I have decided to go out and record sounds to be used in the game. First off I needed to record footsteps on different surfaces so I walked around outside, in puddles, on and even on wooden surfaces. I would have preferred to do this in a Foley Studio but the companies that I contacted didn’t get back to me so I had to improvise. I used my phone to record because I’m not a Triple A studio with amazing recording equipment so I used my phone and did that instead. It was a fun and weird experience I had never done something like that, I did get a few weird looks but the sounds I recorded were worth it.

My plan was to insert some the audio clips so that you could listen to them, but unfortunately WordPress won’t let me insert audio files unless I upgrade to a premium membership. I did try and look at ways around this but nothing seems to be working even if I do encode the audio, but you’ll be able to hear it in the prototype.

Unreal Production

09/12/2016

Using all the information I have found, I will start building my scene in Unreal Engine 4. RealSpace3D comes with some really cool sound stages which are like these rooms that already have reverb properties integrated into them; so it should be interesting to see what we can make using them. First I had to set up RS3D in Unreal seeing as it is a plugin, what I didn’t take into account was how difficult it would be. It required me to place the core folders deep into the directory of UE which are confusing to navigate through, afterwards I found out that RS3D doesn’t use blueprints to run, it uses C++ scripts which I luckily know a bit about thanks to college, so I had no trouble creating script which would allow it run.

This is where the errors began, so in order to get RS3D working you had to exit Unreal and manually go into the script class you made and build the level from there so RS could work. But when I tried opening up my Unreal file this error message kept popping up and I tried rebuilding several times and even from the source directly, which is in the plugin directory. Next, I tried to look online for tutorials and guides on how to solve the problem I could only find one thread on RS’s website regarding this matter. I followed everything they said to do and I still got the error message.

So this is the script class that I had to build, I didn’t write the script it was pre written, but I did have change the reference on line 8 and then build the script into my level which took an hour to do. But even after all of this I still couldn’t get RS to work no matter what I did so in the end I decided it would best to leave this alone.

26/12/16

After I chose to work straight from Unreal only by using different blueprint methods and those nodes I mentioned previously. I started off by using the third person template in Unreal from there I got to work, first I made it into a realistic first person view using the blueprints and camera shake blueprints; I could’ve just used the first person template but that doesn’t have the same effect. I started mapping out the first area, here you can walk on different tiles that represent different surfaces and the sound will change depending which surface you’re walking on. I have 6 platforms which are Grass, Water, Glass, Cobble, Wood and steel. I first went into the project settings and started to create physics for the different platforms so my player’s footsteps could react to them, next I created separate Physical materials for each of the sounds and then referenced them in the appropriate material, this way the game knows when the player is on a specific surface and then it knows which material and sound to reference.

Afterwards I got started on the blueprints, this was a lot easier than I thought it would be. All I had to do was first add a notify in the character run anim for when his foot touched the ground, this is so I could reference it in the blueprint. The main node running it all is the LineTraceByChannel which basically has an input from the player actors and footstep notify, this tells it ignore other impacts and focus on the footsteps. The output references the project setting and tells it to set a trigger for the physical materials and for return value to be a sound. That’s pretty much it for the blueprint side, everything is now working and I just have to play around with the sounds.

So next I had to use Sound cues in order to get the right pitch and randomisers for the audio clip. So it’s really basic you just get your sound assets, here I had three different types of grass sounds which ranged from 1 to 2 seconds long, you feed them into a randomiser to mix it up a bit then into a modulator so you can affect the pitch and volume and then into the main sound. That’s pretty much it you just have to do that for all of the sounds. In the documentation it mentioned that you can actually adjust the speed of the sound in accordance to the player’s footsteps, so to do this I had to create a sound class which I could then reference in the sound cues, this so the engine can compile all the sounds and so you don’t have to do it for every separate surface. I then created a sound mix which enabled the sound class to be referenced in the blueprints and also to reduce the volume on the sounds when he walks on the surfaces.

11/01/17

Okay so I’ve finished making all the sound cues and finished referencing all of them as well so everything in the footstep is ready, next I’m working on the 3D sound room in which I will hopefully have a nice ambient music some outdoor scenery.

This didn’t take too long to do, I basically made a small outdoor scene and then I used a sound cue of some birds tweeting and wind blowing, I placed it in the middle to have it as the focal point of the area. It was a lot easier than I thought it would be because when you enable ambient sound Unreal does most of the work for you, all’s there left to do is configure the settings for it. Luckily mine didn’t need much tweaking besides the size of the sound cue so that allowed me to move onto the next area which would be a closed room playing a song in a 3D atmosphere.

Okay so I have started working on my sound room and I have the room set up, but a problem has come up. Whenever I try to set the audio file to have Attenuation and Ambient sound it does not play in game, if I uncheck those boxes then it does play but throughout the level so I’m going to see how I can fix this in the settings, if not then look for the answers online. So I tried swapping the sound file for a default sound file and then that one worked with the ambient sounds, at this point I’ve gone to the Unreal Forums for answers. Ok so I tried to do one last thing which was to turn the audio file into a sound cue and that ended up working, it was fun to adjust the parameters for the ambient 3d sound because in the end I got an amazing result. So now the sound room is finished I just need to do one more sound room which an elevator system so you see how height affects the audio.

So I basically used the sound room as a basis but added an elevator system so you see how height also affects the sound and ambience of the scene. I used a song to show how the sound changes and I also changed the 3D parameters up a bit from the previous room in order to deal with the height sound.

These are the two sound room, there is a wall in-between but I removed it for the purpose of this log.

16/01/2017

The next problem I had was building the actual game, the build kept on failing and it was due the a modifier I used called LandscapeAutoMaterial which wouldn’t let the engine build the game. So I followed some tutorials and they told me delete some folder, big mistake because that wiped all of my game and it wasn’t recoverable so I had to do the entire project. It didn’t take too long to do because all of my assets and recordings were saved and also I knew what I was doing so it worked out in the end.

20/01/2017

Prototype

This is a link to download my prototype

(Note: Please run in Unreal 4.14 or higher)

Click Here!

Video

(Note: Watch with headphone for full experience)

Specialism Subject: Audio Evaluation

19/01/2017

At the start of this project my question was ‘How is audio used to create an immersive experience’. Did I find an answer to this question? Yes and no; I’ll explain what I mean by this throughout.

So first was the research part of the project which was also one of the most interesting. I found out so many eye opening facts and tricks used by industry professional’s in audio and the world of Foley art. The research was the most important part of the entire project because it helped me understand how audio is implemented into games and how that there’s more than one method of achieving it. I actually found a plugin for Unreal which gave the audio a x10 boost, it was called RealSpace3D and it was an audio engine which worked inside Unreal or Unity. I really did my research into this plugin because it looked fascinating, but in the end it did not work because the C++ script kept failing to build the plugin so that resulted in it not working all together. I was disappointed because I did want to explore RS3D in more depth, but I didn’t want it to hold me back so I decided to make the prototype entirely using Unreal. This turned out to work better than I could I ever imagine, the Unreal Documentation was a real help because it had every node outlined and what you could co-use them with. that really helped me connect my blueprints with my sound files.

The production was an enjoyable experience because I learnt a lot about how audio is implemented in the game and I also got very familiar with the technical aspects of it. I have hit some bumps along the way, RS3D not working was a major setback because one of the main aspects of my prototype relied heavily on the plugins power. But it ended up not working which was a setback, but I managed to continue with my work because I wouldn’t let that hold me back and Unreal really did end up surprising me because it’s audio aspects worked really well. Obviously not as good as RS3D would’ve done, but it was still fairly good, the 3D sound didn’t work as good as I wanted it to though. But it all ended up working in the end.

Overall it was relatively fun experience, I learnt a lot of new things and hopefully you did too. There are still more nodes and blueprints for me to learn and because I did not have time to cover some of the ones I originally planned to, so in the near future when I have some time I will be looking to cover them. Things didn’t really go according to plan, but that’s fine because I have now finished with the project and have covered the aspects of audio which I wanted to. If I were to this project again I would defiantly try and get RS3D to work again because that seems like a really interesting program to revisit.