A Postmortem on Implementing AutoPlay in UE4 | Unreal Fest Europe 2019 | Unreal Engine

Just another WordPress site

A Postmortem on Implementing AutoPlay in UE4 | Unreal Fest Europe 2019 | Unreal Engine

>>David Sim Wei Lun: Hi, everyone Thank you for attending this last talk of the day As you can see, this session is “A Postmortem on Implementing AutoPlay in Unreal Engine 4.” To preface, as you can probably tell, I cannot talk about the project directly, other than it is a AAA Action RPG for PlayStation and Xbox One It might be a little obvious what the project is, but I have a little request, which is, if you were to make any social media posts, do not directly associate this talk with the project itself Unreal Engine 4 will be referred to as “UE4.” For development of this project, we used UE4.17.2 First, the contents of this talk, there are three major topics that I will go over To start off, a brief self-introduction, followed by some information on the automation work I have done on this project Next, I will talk about the AutoPlay system Finally, I will go over the results of using said system I have made the talk such that it should be understandable by non-programmers, but there is a lot to go over, and there might not be enough time for Q and A, so feel free to approach me afterwards if you do have questions Anyway, let me briefly introduce myself My name is David Sim Wei Lun, but please call me David I was born in Singapore, and joined Square Enix in June of 2017 Before then I had worked as a programmer in Singapore and in America For this project, I mainly worked as an automation programmer, but aside from automation, I have also done work related to Engine, tools, player controls and enemy combat As far as automation work goes, in addition to Auto Play, I worked on automating a handful of other systems Broadly speaking, these systems can be broken up into three categories The first type involves updating data through the Editor This is done using UE4’s automated test feature Most of you probably know about it by now, especially if you attended Jessica Baker’s talk yesterday, but just in case, you can access this feature through UE4’s session frontend window, under the Automation tab Using these features, various unit tests can be run on the Editor, the Engine, the game, and so on The automation tests I implemented were mainly for the environment team This included tests that automatically updated existing Asset types, like HLOD, as well as custom data Assets related to permanent resident Assets and Texture loading In general, the tests would reference some existing data, and either import or update the target data For example, when updating the HLOD data, the test would open and reference the target map After updating, it then submits the data to the version control system, or VCS, that we use Moving on, the next type of automation is for quality assurance This typically involves profiling systems, but can include other in-game system, such as for the system to play all cutscenes It is quite a broad topic, but the general flow of automation for each system is quite similar In general, each system has their own special launch arguments that our continuous integration system specifies Then, in order to figure out what to load and in what order, the launch system reads that information from a text file Typically, the system is hooked into various events using Blueprint nodes, and controls the game using debug commands, while the actual profiling is being carried out As far as the commands and nodes go, we used both pre-existing and custom made ones, while profiling results are usually simultaneously updated to a database and output to a .csv file Once the profiling is done, our CI system gathers the results and performs its own upload to a file server, before finally sending a notification email out To be precise, there are three kinds of QA-related systems that we automated The first includes a system that measures the performance of Actors, one at a time, in a blank map There are about a hundred Actors, including players, party members, and enemies Well, it was only for enemies — there was another system that profiled roughly 450 attacks The second category involves profiling various maps Initially, it was only 12 maps, but that eventually expanded into various locations within each map itself, and that grew from 50 to about 720 locations in total The last category was in so much use for profiling

than it was for creating cutscene-related data One such system had to play approximately 270 cutscenes in succession Regarding automation of this kind, the bottleneck had to do with the scope of the data, especially when it came to the large number of locations and cutscenes While we wanted to run these systems daily, a single run would often take more than 24 hours This made it impossible to run it within a day, especially after including the downtime from error handling, and starting up, and cleaning up In the end, however, we came to the solution of being able to run it daily by splitting the work across two to four Dev kids Another bottleneck was that if a crash and error, or for some other reason, the system were to fail halfway, it would not be able to resume profiling the remaining data by itself To solve that is the third type of automation, which involves external tools While this type is not just launcher tools, per Se, launcher tools were used for handling potential errors that may occur while running profiling systems, especially at the scope that we were running them at The tool itself is not that complex, however It essentially uses our target platform’s API to launch the game and monitor its output log If an error is detected, or if the time limit set is exceeded, the tool would restart the Dev kit and have the profiling system resume where it left off Other tools under this type include a tool to analyze cook log warnings, one to organize the profiling results, and another that merges data for PGO, which is a type of automization Since these tools are external, they are not directly related to UE4, and you can say CICD services also fall under this category Usually all three categories of a system were used together as a set; typically the external tools managed in game systems, which in turn generates various data The Editor then uses this data to update various Assets It is using this flow of automation that automated the generation of PGO and cutscene-related data As you might have already guessed, AutoPlay can be considered to be an in-game system Before I go into details of the system itself, let me briefly go over AutoPlay as a concept As the name implies, AutoPlay is a system that can play through the game automatically I think there are various ways to go about this, of which I will give three fairly simple examples The first example is by replicating the user’s input exactly This is probably one of the simplest methods, and so the more complicated the game gets, the harder it might get to use it efficiently Since the input is replicated at pre-determined timings, it is easy for the replication to fall out of alignment with what is actually happening in the game, especially if any lag or loading time fluctuations occur For those same reasons, this method is not very compatible with the non-deterministic parts of a game For those unfamiliar with what that term is, I will get back to the meaning of determinism and non-determinism later in this talk The second example is by using AI, or in particular, randomly moving through a space by referencing in-game data, such as a Nav Mesh As opposed to progressing through a game in a linear fashion, it is a method that is well-suited towards exploring an area thoroughly, to check for things such as collision issues, or performance The third example is by using some other kind of data to move through the game’s World on a fixed path Some refer to this as “Golden path.” For example, representing the data’s coordinates, and while moving between coordinates, performing certain actions at a fixed time By the way, as far as resources go, I had drawn inspiration from Bayonetta 2’s AutoPlay system, and the Yakuza series auto test system I reference various resources, including those that I have up on the slide In a sense, this version of AutoPlay is a hybrid of all three examples However, the third example is probably the closest to what the system is at its core In other words, it is a system that primarily uses coordinates to move along a fixed path, but also uses input replication and game data to support its progression through the game I will show a longer version of this at the end of the talk, but this is what the system looks like in action To be more specific, what the system does is,

reads the coordinates from a text data file, and move the player along a fixed path of those coordinates, without the use of AI While moving between coordinates, the system replicates user input, as well as executes commands, both of which are specified within the text data as well When the data alone is not enough to overcome things like special situations or minigames, the system then relies on in-game data to advance That is the basic idea behind the AutoPlay system But as for how I decided on the system’s design, I will explain that by going over the five objectives we came up with, while also going into further detail about the actual systems themselves By the way, the objectives are numbered, but it is just so that they will be easier to follow, and they do not reflect the actual priority To start off, one of AutoPlay’s objectives is to save time on playthroughs That is simply because due to the scope of this project, a single playthrough is very time-consuming Towards the end of development, a single run typically took over 10 hours This seemed like a huge waste and an inefficient use of time, especially when key developers were involved Admittedly, I also have my own personal interest in such a system since the time I was in America As mentioned earlier, also inspired by other titles that implemented similar systems Three systems were implemented to accomplish this objective; a recorder system for creating the data, the AutoPlay system itself for running the data created by the recorder, and a play list system for running multiple data consecutively As I was making use of my experience with the other automation systems on this project, the foundation of the AutoPlay systems bear multiple similarities with them For example, all these systems require specific launch arguments and text data They also attach to the same callbacks, and control the game using more or less the same feedback commands In addition, they are all updated around the same time, which is near the end of the game instance’s tick As long as the arguments and data are valid, these systems can be run on any Dev kit, which includes both PS4 and Xbox One However, one tiny difference is that the AutoPlay systems are mainly run using the command line Moving onto the next objective, which I that the data can be created and run by anybody This is mainly because it was not realistic to handle the data creation all by myself, given the scope of the project It also seemed more cost-efficient to build the system in a way, where I was not limiting the potential users of the system In the end, we went with having a three to four person team, nicknamed the “Data Team,” to handle data creation and support It is thanks to this team that we were able to finish making the data For data used with PGO data generation, however, we did have another person handling it It is based on this objective that the recorder system was implemented In order to keep the bar of entry low, plain text was used for the data for that We did consider using behavior trees and AI controllers instead, but decided against it since that limited the number of potential users What the recorder mainly does is, record gamepad input It also has a debug input node, where instead of recording input, it records player coordinates, or changes the recorded settings based on specific button combinations As shown on the slide, the recorder outputs coordinates in this format, line by line If any buttons were pressed prior to recording the coordinates, it would also output the timings and buttons in the order that they were pressed The coordinates and timings are written as numerical values, and to make processing easier, a single character is used to represent each button There were also various commands that can be specified, instead of button input Commands can do a wide variety of things, such as making the player wait a specified amount of time, or skip to the next set of coordinates, or receiving a cutscene notification, and so on The more elaborate the command, the longer they get,

and are not usually limited to a single character The downside is that most of these commands have to be input into the text data by hand; therefore, the typical workflow for creating the data is to first create a foundation to work with, using recorder, then through trial and error through running the data, perform adjustments or add commands by hand until the data is complete Going back to the topic of objectives, another is to handle deterministic sections with data Conversely, it is also to handle non-deterministic sections with code Just in case, let me briefly go over what I mean by deterministic and non-deterministic A process can be said to be deterministic if, given the same parameters, it will always produce the same results On the other hand, a non- deterministic process’ results cannot be predicted accurately, even with the exact same parameters each time If a deterministic game, we could, in a sense, recreate any given frame within a game, as long as we know the parameters of that frame Chess and certain fighting games can be said to be deterministic, as well as the previous talk about RTS games In my opinion, the more interconnected action there is in a game, and the more random elements there are at a given time, the closer the game gets to becoming non-deterministic As you may imagine, while there are deterministic sections in this project, there are a lot of non- deterministic sections as well, so the use of an input replication system was simply not enough to achieve our goal, of being able to play through whole game automatically We relied as much as possible on data, particularly for deterministic things, like linear movement and fixed timing input, but for non-deterministic things like battles or minigames, it was better to handle that with code, and let the AutoPlay system take the reins Moving onto the fourth objective, it is to keep the system’s design and features simple This is merely because there was not much development time left in the project One compromise that was made based on this, though admittedly not a very desirable one, was that it was okay for the AutoPlay system to take up some performance Based on this objective, priority was given to tasks that could be more easily or efficiently solved Harder tasks were temporarily solved using debug commands, and were eventually dealt with properly For example, we would skip minigames by warping the player to the point after they were completed, or use a cure-all command to win battles About the actual details of the AutoPlay system itself, I will start with going over how movement is done At its core, it is a system centered around coordinates-based movement; that is to say, it moves the player along a fixed path I do think, however, that an AutoPlay system that uses AI for random pathing could be a great complement to this system You have already caught a glimpse of it earlier, but this is what it looks like when the system is moving the player As you can see, it is simply moving the player in a straight line, in between coordinates And while moving between points, the system performs other inputs at specific timings Regarding the implementation for carrying out input, it is done through the PlayerController to make it as close to the cull path of actual user input The camera direction and the direction of the destination relative to the player are used to figure out which direction the left stick should be tilted towards The calculation itself is actually not very complicated, and some of you probably know about it But it is probably easier to understanding, using a top-down perspective If we rotate our view, such that a camera’s direction is lined with the vertical axis off a 2D plane, the direction of the destination will match the direction in which we should tilt the left stick To put it another way, by figuring out the angle between those two directions, we can rotate to the up-vector by that same amount to get our desired results Movement is relatively easy, but handling the various states of AutoPlay is somewhat complicated While in the beginning there were only five States,

which was not complex at all, in the end it grew to a total of 31 States, from handling various menus to all kinds of minigames and Game Modes Having to deal with them individually made the process complicated This is especially because certain States have the possibility of being interrupted by other States, so a lot of care was required to handle each potential transition properly Also, while debug commands were used to temporarily skip the non-deterministic parts of the game, they will eventually handle properly through the AutoPlay system, by looking up information how the relevant managers or Pawns For example, if the enemy manager Class showed that there was at least one enemy, the AutoPlay system would switch to its battle State Or if the UI manager class showed that it is displaying the title screen, the AutoPlay system would initiate the necessary button inputs to start a new game One other thing this game had a lot of is what you would call special situations; basically, anything that is not part of the core gameplay loop, including minigames or special gameplay modes While the way these were handled are by and large the same, there are roughly three types of these situations The first type involves using a specific ability of the players in order to progress further in the game For example, pointing the camera at a specific location, then activating a certain ability in order to progress to said location For handling such cases, using the information stored in the PlayerController or Pawn was sufficient The second type involves either controlling an Actor that is not a main player character, or navigating specific UI There is a possibility that the controls may be different in such cases, but usually it is enough to simply switch to using the relevant Controller or Pawn The last type involves minigames Typically, player control is retained, but the objectives are varied in nature; such as collecting a large number of moving targets, or having to press a randomly selected button For such cases, they are handled by searching for the target, using either a Pawn or Actor iterator, and by checking for obstacles using line traces The AutoPlay system also has what is known as a “bug detection and tag feature.” The bug detection features checks for three things; whether the player’s Z-coordinates is near the Kill-Z plane, whether the travel time between destinations has exceeded five minutes, and whether the total time passed since the data was run has exceeded a certain amount of time This is to check for any game-related bugs, such as the collision settings of the environment Another thing this checks for, however, is whether there are any issues with the AutoPlay data itself This is because the game was changing at a high frequency, and there were a lot of non-deterministic sections, so it was often the case that a system would get stuck at various points of the game, at least until development had slowed down Regardless of which bug it is, if any of the three checks failed, the system would output to an error log, take a screenshot, and optionally create a dump, so as to make investigating the cause of the failure easier In order for the AutoPlay system to continue executing the data with little interruption, after outputting everything, the system would walk the player to the current destination and target the next destination As for a tag feature, it is used to associate a debug jump point with a destination If the system sees a tag associated with the destination, it will use the tag’s jump point instead of warping the player The tag feature is also used for executing the data from its middle, though that is not related to the bug detection feature itself Even if the system were to get stuck in dozens of places, using the bug detection and tag feature, we would typically be able to resume data execution Overall, this allowed us to maintain coverage of most of the game’s playthrough In a sense, that was another feature of the AutoPlay system Its default behavior is quite simple We have a list of enemies stored in our enemy manager class, which we target in order When I say target, I do not even mean locking onto the enemy; the player simply makes a beeline to the target, and once within a certain distance would spam the attack button

There are not any path calculations or obstacle detection, so it is possible to get into situations where the player is unable to carry out the single attack; for example, if the player were to get stuck on an environment, or if the enemy is larger than the range used for determining whether to attack To counteract this, after a certain amount of time passes, the system would use a debug command that kills all enemies To prevent Game Overs from occurring, another debug command is used to make the player invincible In addition to this really simple battle mode, an expanded version was made mainly for PGO This version makes more use of all the features available in battle By making use of UI and player information, this version will lock onto enemies, switch targets on the fly, and execute any of the abilities available to the player at random By using this mode, we applied it to performance-intensive battles and player abilities to generate PGO data, in order to perform various optimizations There are two other features of the AutoPlay system; the loop system is as its name implies — when execution of the data has finished, the system loops back to the beginning of the data and repeats it The retry feature is also kind of straight-forward Doing battle, it would set the player’s HP to 1, and deliberately control the player in a way to cause a Game Over It then selects the various options available in the Game Over screen, and resumes data execution from the previous destination The intent of this feature was to test the many checkpoints in the game that were used after a Game Over Unfortunately, this is one of the only features that was unused, due to a lack of time, and updating the AutoPlay data to be compatible One final thing about the AutoPlay system is how it is launched First off, as mentioned before, specific launch arguments are required to enable the system Once enabled, it can be activated through the command line using the AutoPlay keyword, followed by the name of the data file Various flags can also be specified after the file name; for example, how many times to loop the data, whether cutscenes should be skipped, or how the system behaves during battle The final system implemented, which is a level above AutoPlay, is the List system In a sense, this system ties together all four objectives mentioned thus far As you can guess from this name, the List system’s primary feature is to execute AutoPlay data consecutively, kind of like a music playlist I should also mention that we organize the AutoPlay data on the per World basis, so each file represented a playthrough of a single World The list data itself also uses a plain text file which, as shown on the slide, lists the data files in the desired order of execution The AutoPlay flags can also be specified separately for each data file, which will only apply to that file during execution Similar to the AutoPlay system itself, the List system is also activated using the command line and its corresponding keyword, followed by the file name This system can also be started in the middle or looped In addition, you can specify AutoPlay flags on the AutoPlay List command line, which would override the flags within the list file With regards to the loading of the data and list files, it might be worth mentioning that it is all done using pre-existing FFileHelper and FString functions Last, but not least, is the fifth objective, which is simply to use the build types that have debug features The test build was made the primary target, since it retained certain debug features while being the closest to the release build Whenever we needed additional debug features or information, we will use the debug build instead While we did consider using the shipping build, it did not have much debug features, or if any at all, and it seemed risky to enable something like AutoPlay on it We decided against it in the end I will next talk about the results of AutoPlay, but before that, let me briefly go over its implementation costs, and how it was actually operated It took roughly two months in total to implement all three systems from scratch It took an additional month for the systems to be considered feature-complete,

which includes things like being able to properly handle very special situations, and the expended battle mode However, what is not included in the time frame is the additional support costs for AutoPlay What I am including in said costs are things like having to update the data, fulfilling various requests, and fixing bugs Including the efforts of the Data Team, this lasted all the way to the end of development which, I would like to emphasize, as being rather costly Regarding the data files, in the end there were 20 of them in total This includes files for each World and files for progressing in between the Worlds There were 400 total revisions across other files, so each file had taken roughly 20 revisions Based on the graph shown, it took on average six months of support per data file It is a little hard to tell based on numbers alone, but most of the support costs came from finding issues of the data on a daily basis, and fixing those issues However, I should also mention that both the Data Team and I had other responsibilities, so it was not like we were working only on AutoPlay all the time Next is the operational workflow of AutoPlay As you probably expect, data-related tasks were mainly handled by the Data Team, while system-related tasks were handled by me First, the Data Team creates the necessary AutoPlay data Then both the Data Team and QA runs the data each night before heading home The kits that the data was run on include both PS4 and Xbox One I also run the data on my kits whenever they are not being used for other tasks Then, on the next day, everyone uploads the results to a file server The Data Team then checks those results, and reports any findings through chat and-or email If there were any issues, we would first verify whether it is an actual game bug by contacting the developers related to that section of the game, and attempt to reproduce the bug by hand If it appears to be an actual game bug, we would create a bug ticket, which QA would then verify If they are able to reproduce it, we would then escalate the ticket accordingly If it is not a game bug, it is likely an AutoPlay bug We would then verify if it is data-related or not If it is, it becomes a Data Team task Otherwise, especially for things that cannot be fixed without code changes, it would become a system task on my end We also have some basic rules regarding submitting data to our version control system Basically, we only allow AutoPlay data used for playthroughs and PGO to be submitted There are a whole bunch of other smaller data, which I created for things like testing bosses or reproducing crashes, but those were never submitted All of the data, submitted or not, were placed in a game data debug folder, since it would never be included in the release build This directory is also where we placed the text data used by the other automation systems Moving on to the results of AutoPlay, I will start off with how it benefitted the project There were three main areas where AutoPlay was used One area is, as you can imagine, QA-related As expected, we saw the most usage out of playthroughs, but it also saw usage in checking environments and he behavior of bosses The Sound Team also made use of it for audio-related checks Another area involves checks over an extended period of time This mainly has to do with bug reproduction and verifying bug fixes, especially the kind that have such low reproducibility that it took an entire night, or even days, to verify Towards the end of development, AutoPlay was also used for investigating memory leaks The last area has to do with optimization In addition to being used to generate PGO data, AutoPlay was also used for observing the game’s memory, load time, and Texture pull By tracking those elements of the game, we were able to discover other types of memory leaks and irregularities with certain cutscenes Relative to the initial goal of having a system that can carry out play throughs, we saw a much wider usage than expected, which is awesome Even though there are only a few users per team, there were about eight teams in total that used AutoPlay,

which is more than we had expected Furthermore, by running AutoPlay every day, we were able to discover or confirm multiple hard to reproduce crashes Also, this is a very rough calculation, and it might not mean much, but in terms of the total amount of time that AutoPlay was used, which excludes any estimated downtime from bugs or breakages, it amounts to approximately 900 work days, across all the teams Of course, as useful as the system may have turned out to be, that does not mean it does not have any shortcomings Once such issue was its support cost It is precisely because of the system’s simple design that it became tricky to deal with new aspects of the game that were added later in development That led to a significant amount of tech Dev, and because of the high frequency in which the game was updated, it was only towards the end of development that we started seeing greater stability in the AutoPlay data Editing the AutoPlay data is not easy, either The system’s documentation is really long, due to the number of features and minute details it has, and there are a ton of commands that can be used, which made it challenging, even for me, to edit the data that was doing complex things There are also a lot of possible flags, so if you are using some complicated settings, entering all of that into the command line could be quite tedious Furthermore, since AutoPlay does a bunch of atypical things, such as running various debug commands depending on the situation, even if we discovered what appears to be a bug, it was not always obvious whether it was actually a bug of not To be fair, however, this issue was somewhat alleviated after we started using capture cards to record everything, since we could better analyze the circumstances that led to the potential bug One other thing which is not so much an issue, but rather an unfortunate side effect, is that because the system did take up some performance, we did not really use it for profiling Most of this performance came from carrying out various checks on a game, every frame on the main thread While this was avoided as much as possible by relying on attaching to callbacks instead, due to the potentially complicated nature of State transitions and non-deterministic sections, there were a lot of times where checking every frame could not be avoided We, of course, would like to tackle all of these issues over time For reducing the high support costs, I think putting more emphasis on planning out the system would help While it might be fine to simply increase the dependency of AutoPlay’s data on the game’s actual data, that may simply be shifting the burden of responsibility to the people responsible for set game data, so I think this is quite a delicate balancing act It more or less boils down to a question of who should shoulder the cost of supporting the data and system, which should be decided on carefully, early on That is pretty much what we did for this project, and it is how we decided on the Data Team members For reducing the difficulty in editing data, I think abstracting the obtuse parts away with an Editor tool and reworking the data format would help Similarly, for reducing the time taken to put stuff on the command line, switching to a GUI should not only help, but also make it easier to control the system To be honest, I have yet to come out with a decent solution for bug differentiation, other than continuing the usage of capture cards, and possibly improving our log output As we are using the system for profiling, I am hoping to switch to a multithreaded design, or possibly converting the system into an external tool This is a really rough compilation, but there are plenty of other improvements that we are also looking into, which range from expanding the system to further automating its operational workflow To summarize this postmortem, through the implementation of three systems, based on five objectives, we were able to fully automate playing through the main story of the game These goals were through collaboration with the Data Team, resulting than much wider usage than expected While the support cost was high, we saw high returns as well, which may actually be greater than our conservative estimates There is also plenty of room for improvement,

and by maintaining the momentum behind such a system across multiple projects, I believe AutoPlay can become an indispensable system for the development of any AAA title I will just let this run to its end, which is almost there That is pretty much it Thank you for your time today Thank you for listening Thank you [Applause] ♫ Unreal logo music ♫