Unity3D is one of the most widely used engines in the game industry, several extensions have been implemented to increase its features in order to create multimedia products in a more effective and efficient way. From the audio development point of view, Unity has included in later versions an Audio Mixer, which facilitates the organization of sounds, effects and the mixing process in general; however, this module can be manipulated only through its graphical interface (GUI). I designed and implemented an extension tool to map parameters from the Audio Mixer to MIDI external devices, like controllers with sliders and knobs, to allow the developer to easily mix a game with the feeling of a physical interface. The tool can be purchased from the Unity Asset Store.
Unity Audio Engine
The Unity Audio System supports most standard audio file formats, plays sounds in 3D space, applies filter effects to those sounds, records audio from a microphone, and more specfic features for creating
an immersive experience.
The audio system works as follows: Sound files are represented by Unity as Audio Sources, that can be placed in the game scene to reproduce audio. These sounds are received by an Audio Listener, which is a reference point of hearing; that is, audio will be played in 3D according to the position
of Audio Sources in regard to the Audio Listener. Audio Filters; such as echo, reverb, distortion, and others, can be used along with the sources, or a set of them, to modify the environment.
Unity implemented an Audio Mixer for mixing, applying effects, and perform mastering. The layout for this tool is shown in the image below. The mixer consists of Groups, Snapshots, and Views. A Group is a mix of sounds that controls Audio Sources as long as they are linked to that group. Volume attenuation, pitch correction, and effects insertion can be manage from a group. A Snapshot saves the state of the values of groups and it is useful when a whole mixing changes according to a location. A View is a graphical arrangement of the groups; that is, one or more groups can be hidden and that presentation can be saved as a view, which is useful for avoiding being over whelmed when there are lots of groups.
In the top-right corner in the previous figure, there is a drop-down list called Exposed Parameters. They are mixer parameters like volume, pitch, echo delay, and any other parameter attached to one or more groups. Unity is able to control these parameters by exposing them to script control for in-game mode. Thus, the value of the exposed parameters can be changed in real-time inside the application; also, they can be manipulated in the audio mixer directly using the editor GUI, and then be saved as an snapshot in editor-mode; however, an editor-script control is not possible, unless it is in the in-game-mode.
System Design
Mapper Overview
The main goal is to map the parameters of the Unity Audio Mixer to a physical MIDI interface in order to control them in an easier way. Unity does not support some features that are necessary
to implement the mixer mapper, such as MIDI communication and modification of exposed parameters through scripting in editor-mode.
To include MIDI communication, the library NAudio was integrated to Unity as a plugin in order to support MIDI features in Windows. NAudio is an open source .NET audio library, which is compatible with the Unity scripting language C#.
To modify mixer parameters through scripting, in editor-mode for real-time mixing, it was necessary to expose them; however, the modification of values for exposed parameters only works in the in-game-mode and are not saved when the application stops. To solve this problem, the mixer mapper modifies the values directly in the audio mixer file, hence the modifications are not lost. This strategy needs to parse the audio mixer file whose format is YAML, a data serialization language that is used by all Unity file.
The system overview is shown in the next figure. The developer uses the Editor Mapper GUI to assign MIDI controls to exposed parameters in the audio mixer; then, the MIDI controller can be manipulated to change the values of the parameters. The core of this process is the Audio Mixer Mapper in which all the logic resides. The mapper allows the user to match the parameters with the physical controls
and save them permanently, by modifying the audio mixer YAML file through a YAML Parser module. In this way, the exposed parameters have a direct influence in the Audio Sources from the application according to the audio mixer management.
Editor GUI Design
The next image shows the graphical user interface (GUI) for the mixer mapper. This custom interface was implemented following the standard layout of Unity editor windows. This image presents the main sections for the mapper: the target mixer, which is the audio mixer that contains the exposed parameters to be mapped; MIDI devices that shows the controllers connected to the computer where Unity is running; the start snapshot, which is the configuration that is going to be mixed; and, the exposed parameters, represented by sections with relevant values for the parameters to be mapped.
Depending on the state of the mapping process, an exposed parameter section can have one of the views presented in the figure below described as follows:
- Unmapped: There is no MIDI control associated with the parameter until the user press \Learn” to assign a control.
- Learning: The tool is waiting for the input from a MIDI device; that is, if any knob, slider or other kind of control is moved in the device, it will be detected and you will see the green bar moving while you move the control. By pressing the button “Learning…” the user exits from this state.
- Mapped and inactive in mixer: The parameter is now mapped to a MIDI control and it can be confirmed when the control is being moved and the cyan bar is changing. However, this will not affect the Audio Mixer yet because you need to run the application in the in-game-mode.
- Mapped and active in mixer: In this state, the words “Active in Mixer” appears on the top of the section, which means that the exposed parameter is reacting to the MIDI control. It can be observed when the control is being moved and both, the exposed parameter and the mapping cyan bar, are changing. This state only happens when the application is playing.
An exposed parameters section, as shown in the previous image, four parameters are depicted, composed of the following attributes:
- Name: The given name to the parameter when it was defined in the Audio Mixer. (volume master is an example).
- Mapping Type: The type of mapping regarding the MIDI control that depends on a curve. This attribute dictates the behavior of the control and how it behaves when it is manipulated. When a type is selected, the curve can be seen in the field below. The types are: Linear, Logarithmic, Hyperbolic, Quadratic, Exponential, and Custom.. For the implementation of the curves, the derivative of the functions corresponding to each curve was used in order to create the points and tangent lines that compose the curve.
- Curve: The curve that results from the selection of the mapping type. The custom curve enables this Curve field where the user is able to provide any curve by adding points and modifying
tangent lines manually. This is a very flexible feature that allows to test unusual behaviors for mapping. - Device No.: This attribute is the number of the device that is mapped with the parameter. If there is no mapping, it will be 0 (zero), otherwise it will be the number that you can see in the device panel to the left of the main window.
- Channel: The MIDI channel that was learned in the mapping.
- CC: The MIDI control code that was learned in the mapping.
- Min Value and Max Value: These values can be modified and must have the limits of the parameters that are exposed. By default, the values are -80 and 20; it assumes that a volume parameter is going to be controlled.
- Current Value: The parameter that will be reflected in the Audio Mixer. It can be changed manually in order to give more precision if required. To its right side, there is a small red led that twinkles when the parameter is being manipulated by a MIDI device.
- Parameter Bar: This bar is a graphical feedback exclusive for the MIDI device; that is, it reacts to
the movement of the MIDI control that is mapped to the parameter. Its color is green when it is learning and cyan when is already mapped. Also, it can be manipulated with the mouse pointer. - Learn Button: When this button is pressed, the parameter enters to a learning state to catch a MIDI control and hides the rest of parameters.
- Forget Button: This button only appears when the parameter is mapped, and allows the mapping to be released and the controlling stopped.
Demo
A simple demo was developed as an example for the user. An screenshot of the demo is depicted below, which contains three main categories: music, ambiances, and sound effects, where audio sources are linked to the corresponding category, and the mixer mapper is set with exposed parameters ready to be used. Also, a video manipulating a physical MIDI mixer is shown.
This tool was presented in a paper/demo session at the 19th International Conference on New Interfaces for Musical Expression (NIME 2019) and the related paper can be found here.
