Westerdahl System Architect Tools The Asset pipeline for Just Cause 2 Lessons Learned 2 The Asset pipeline for Just Cause 2 Lessons Learned Overview Background Just Cause 1 Just Cause 2 ID: 553303
Download Presentation The PPT/PDF document "Mathias" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Mathias
Westerdahl - System Architect, Tools
The Asset pipeline for Just Cause 2: Lessons LearnedSlide2
2
The Asset pipeline for Just Cause 2: Lessons Learned
Overview
Background
Just Cause 1
Just Cause 2
Current pipelineBuild system and compilersCentral code Take aways Q & ASlide3
3
The Asset pipeline for Just Cause 2: Lessons Learned
What is Just Cause?
Large open world
Sand box game play
Dynamic game play
No Levels, continuous streaming Multiplatform Very flexible editing and testing of content Needs a fast and flexible pipelineSlide4
4
The Asset pipeline for Just Cause 2: Lessons Learned
JustEdit
Our main editor is called JustEdit
Edits the terrain, missions, locations, roads, etc.
Each mission/location contains entities
An entity contains game objects or other entities This highly flexible system led to very many assets to export (~1500 locations and ~4000 entities) Runs the game No turnaround times for testingSlide5
5
The Asset pipeline for Just Cause 2: Lessons Learned
JustEdit
Maya
Perforce
Damage Editor
Terrain
Locations
Missions
Particles
Models
Physics
Animations
Skeletons
Physics
Workflow Overview for Just Cause 1
GameSlide6
6
The Asset pipeline for Just Cause 2: Lessons Learned
JustEdit
Maya
Terrain
Locations
Missions
Particles
Models
Physics
Animations
Skeletons
Physics
Workflow Overview for Just Cause 1
Many manual steps
Game ready data in version control
Game ready data in version control
Editing of game ready data!!!
Perforce
Damage Editor
GameSlide7
7
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons learned from Just Cause 1
Remove as many manual steps as possible
Put export steps in separate tools (i.e. compilers)
Decrease code complexity
Increase robustnessEasier to read and debug Introduce intermediate data formats Tools code broke easilySlide8
8
The Asset pipeline for Just Cause 2: Lessons Learned
Actions Taken after Just Cause 1
We decided on creating our own content build system
Compilers in C++, derived from previous export steps
Took existing code and put into use directly
The Project Compiler was the heart of the systemTriggered all other compilersDependency checking was implemented separately by each compiler (time stamps) Users can right click on any file and compile that dependency tree
The editor and other tools use this functionalitySlide9
9
The Asset pipeline for Just Cause 2: Lessons Learned
Actions Taken after Just Cause 1
Auto Builders were setup
Code
All compilers, exporters, plug-ins, editor
ContentFull nightly builds for all platformsBuilds ISO’s Cruise Control + NAnt was used to create the auto builders
CCTray is a system tray icon tool
Alerts users when a build failsSlide10
10
The Asset pipeline for Just Cause 2: Lessons Learned
Just Cause 2 – New Tools
A lot of new tools were ordered by the content creators
Detailed design docs were written
Each tool was assigned a “customer”
The programmers and the customer kept close communicationThe customer was responsible to test the tool and approve it Tools as plug-ins to JustEdit WTL was used for GUI in editors/plug-insSlide11
11
The Asset pipeline for Just Cause 2: Lessons Learned
JustEdit
Maya
Perforce
Compilers
Terrain
Locations
Missions
Particles
Models
Physics
Skeletons
Compile
Game
Workflow Overview for Just Cause 2
MotionBuilder
Animations
Cutscenes
Constraints
Plug-ins
Vehicles
Characters
Cutscenes
Auto Builder
SyncSlide12
12
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – New Tools
10 new tools at start of project!!
Code not up in the 90%-100% complete where it should be
Flaws in workflows
Not all tools were used in the extent that was intendedContent that was easy to edit was not very suitable in game with respect to performance and memory Design docs were accepted “as-is”They were essentially a wish list from content creatorsSlide13
13
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – New Tools
Customers
They were not part of the dev-team, communication suffered
Often planned full time on other tasks
Unclear responsibilities Plug-insUnstable C++ interfacesShould have gone with network communications
Too complex
GUI: WTL and C++
WTL is too bare bone and C++ slows down iterationsSlide14
14
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – Compilers
Compilers in C++
Using C++ for scripting isn’t always very convenient
Long build times, making it awkward to tweak
Some compilers included all game code Very platform specific Win32 heavy with DirectX amongst other things
Couldn’t run on auto builders with no desktopSlide15
15
The Asset pipeline for Just Cause 2: Lessons Learned
Lessons Learned from Just Cause 2 – Compilers
Programmers implemented things slightly differently
Causing anomalies in overall behavior
Breaking dependency checking
Hard coded parameters in the compilers File formats weren’t stable enoughIf something went wrong, compiler probably crashed
Or worse, the broken data made it into the gameSlide16
16
The Asset pipeline for Just Cause 2: Lessons Learned
Reluctance to work with compilers
The pipeline was considered somewhat magic
Lack of documentation
The overall flow wasn’t easily overviewed
Debugging broken data often done in game codeProgrammers already had all code & data set upThis sometimes led to runtime fixes being done rather than fixing the bugs in the compilersLessons Learned from Just Cause 2 – CompilersSlide17
17
The Asset pipeline for Just Cause 2: Lessons Learned
No central code
Not using central code meant problems when integrating
Fixes went missing, you had to find them again
Built compilers were version controlled in Perforce
Not automatically rebuilt and used when neededYou could still use old buggy codeLessons Learned from Just Cause 2 – CompilersSlide18
18
The Asset pipeline for Just Cause 2: Lessons Learned
Data formats
A lot of different file types (~30)
Many of the formats were Xml based
Mapping “property name” to “value”
Formats were more optimal for the editor than for the gameRead/write errors weren’t detectedNo version numbersNo proper error messages could be loggedLot of things to improve!
Lessons Learned from Just Cause 2 – CompilersSlide19
19
The Asset pipeline for Just Cause 2: Lessons Learned
Content build system
A bit naïve solution
Slow dependency checking
Incomplete dependency tree
Time stamps are not good enoughDue to bugs & anomalies in the behavior, users didn’t trust it to 100%“It went wrong! Did I do it in the wrong order?”“I’d better resort to my old ways without asking anyone…”No trust meant full clean and full rebuild
Lessons Learned from Just Cause 2 – CompilersSlide20
20
The Asset pipeline for Just Cause 2: Lessons Learned
Statistics from Just Cause 2
3 platforms: Win32, PS3, Xenon
~110000 source assets in version control
~80000 of those files pass through our compiler pipeline
Source data amount to ~13GB Compiled data amount to
~7GB uncompressed
~4GB compressed
A full rebuild took about 4h45m for all 3 platforms
Building all tools code took ~1hSlide21
21
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
What was done to address the retrospective from JC2?
Fresh goals
Rebuild the trust in the pipeline
Incremental builds, no magic workflows, easy to configure100% accuracy using the dependency checkingSimplify the developmentMove common code to a central repository
Decrease turn around times when creating toolsSlide22
22
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
Big decisions
We needed a build/deployment system for central code
The content build system also needed a major push
Needed a simple to use and yet powerful generalized data formatDevelopment had to be done alongside JC2Needed to commit to the task to actually get anywhere soonSlide23
23
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Needy
Dependency resolver and deployment system
Written in house using python, very light weight
Handles “packages”
Built libraries (.h + .lib/.so)Built executablesBundles of python scripts (or whatever you want)3rd party libraries / executables
Deploys packages to a central repository
Fetches packages to local system
Dependencies.txt
Python-2.5.150.1013.1;buildtool
Needy-4.4.0.204986;buildtool
Buildsystem-6.1.*;buildtool
Platform-7.*;buildtool
Debug-8.*;buildtool
Adf-8.*
havok-55x.277059
Product.txt
PRODUCT=animcompiler
VERSION=1.0.0
PLATFORMS="Win32 Linux"
CONFIGURATIONS="Debug Release"
CATEGORY=compilersSlide24
24
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Needy
Big change in the coding process
Code base evolves towards smaller libraries
Really fast compile times
Helps keeping track of version conflicts Allows for very rapid development and testingSlide25
25
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline – WAF
Light weight build system for code and data
Open source (BSD), source is ~80kb
Dependency checking
Multi core supportEasily to maintain and extendReplaces our in house Project Compiler systemPython
A small snippet from a task definition using waf:
cls = Task.simple_task_type(‘.anim’, ‘${PYTHONC} ${ANIMC} –i ${SRC} –o ${TGT} –p ${PLATFORM}’)
cls.ext_in = (‘.anim’, )
cls.ext_out = (‘.ban’, )
Task.TaskBase.classes[‘.anim’].scan = FindDependenciesFromAnimationSlide26
26
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF
Avalanche Data Format
Serialization framework
C/C++, Python support
Xml <-> Binary supportHash based versioning for binary filesError handlingThe ability to access data both from Python and C/C++ enables seamless cross language interchange
Cross platform support
E.g. pass data from tools to/from consolesSlide27
27
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF
TM2VertexData = DefStruct( "TM2VertexData",
( "PositionIndex", Array( uint32 ) ),
( "TangentSpaceIndex", Array( uint32 ) ),
( "ColorIndex", Array( uint32 ) ), ( "UVIndices", Array( uint32 ) ) ) TM2Model = DefStruct( "TM2Model", ( "Name", String ), ( "Triangles", Array( uint32 ) ),
( "Vertices", TM2VertexData ),
( "MorphTargets", Array( TM2MorphTarget ) ),
( "Positions", TM2PositionData ),
( "TangentSpaces", Array( float ) ) )
Type definitions:Slide28
28
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - ADF
from Adf import Context
ctx = Context()
ctx.ReadTypeLibrary('mytype.adf')
objs = ctx.ReadInstancesFromXml(‘infile.xml')objs[0].Name = 'newvalue'ctx.WriteInstancesToFile( ‘outfile.bin', objs, BIG_ENDIAN )Read / Write
mytype.py
AdfPack.exe
mytype.adf
mytype.h
compiler.py
Infile.xml
Outfile.binSlide29
29
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Compilers
C++
Shared libraries, loaded from Python
Removed dependencies on old game code
Replaced in-house code with standard codefopen() rather than CFile()Decrease complexity, increase portability
Scripting and logic in python
Data processing in Python or C++
E.g. read/write/process Havok files using C++Slide30
30
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline - Compilers
Python
Extreme turn around times
Vast collection of built in modules
We use these a lot:optparse, ctypes, md5, numpy, cStringIO…Many of the modules have the backend implemented in C++
Most compilers were completely rewritten
Cut a lot of dependencies
Code size down to a 1/10
th
Slide31
31
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline – Dependency Checking
Waf handles the mechanics
Scanners find the dependencies
Waf caches the result
MD5 checksumsCommand lines argumentsSource/Dependency file contentsA new compiler path will trigger a compilation
Very handy when updating packages with NeedySlide32
32
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline – Dependency Checking
A small example of scanning a file for other file references
# Reading a text file containing file references
def FindDependencies(path):
f = open(path, "rb" ) lines = f.readlines() f.close()
return [ x.strip() for x in lines ]Slide33
33
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
Platform independence
Possible to run on win32, linux, ps3, xenon etc.
Helps keeping the code clean
Running on linuxWe wanted to run on linux for the file system performanceRun win32 exe’s through WINE
Wine is a Win32 implementation in linux. It’s not an emulator.
Intermediate solution before all code was rewrittenSlide34
34
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
Incremental auto builder
Syncs from version control
Only builds the things necessary
Full auto builderCompares incremental build with full buildFinds dependency problemsFind bugs in data formatsSlide35
35
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
Change of mindset
Keep it simple!
Each component should do one thing only
…and do it well!Slide36
36
The Asset pipeline for Just Cause 2: Lessons Learned
Current pipeline
To sum things up so far
We rewrote ~20 compilers (out of ~30) in 6 man weeks
Using old code as blue print
Remaining compilers took longer due to more complex work flowsCode size decreased to 1/10thPerformance increased in all cases
10% to 1000%, 100% was quite common
Possible to use compilers in the previous pipeline with little effort
Early benefits of the new pipelineSlide37
37
The Asset pipeline for Just Cause 2: Lessons Learned
Future improvements
Reloading of assets in run time (in progress)
Virtual File System
Can load data from hard drive, archives or web server
Distributed buildsLooking into using AFS, a distributed file systemssh builder01 modelcompiler.exe -i mymesh1.tm …ssh builder02 modelcompiler.exe -i mymesh2.tm …Slide38
38
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
A few thing we’ve learned so far
Some are quite trivial but also easy to overlookSlide39
39
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
Hardware
Defrag the hard drives
Try using other file systems, ext3, ext4, afs.Our pipeline is a lot faster on linux than on win32Two separate discs that are designated source/targetRam discs/SSD are getting cheaperSlide40
40
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
How to implement the compilers
No assumptions on working directory or any other variables
Keep the compiler data driven, it’s quite easy
Project specific options could be read from a file or command lineCreate a simple plugin systemExtremely easy in pythonSlide41
41
The Asset pipeline for Just Cause 2: Lessons Learned
Example of loading python files. I.e. creating a plug-in system:
for root, dirs, files in os.walk(‘c:/myapp/plugins’):
if not root in sys.path:
sys.path.append(root)
for filename in files: if not filename.endswith('.py'): continue # We do an import() instead of an execfile() so that
# the loaded code can act as a regular python file
module = __import__(filename.replace('.py', '') )
Take AwaysSlide42
42
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
How to implement the compilers
Write portable code
E.g. remove DirectX, Win32 API
‘-e’ for endianess rather than ‘-p’ for platformBeware of case sensitiveness from the startAlso, beware of ‘\\’ versus ‘/’Slide43
43
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
How to organize the data
Keep source data separate from compiled data
When creating new file formats
Avoid doing it “by hand”. Create a generalized file type.Include a version numberStore file dependencies easily accessible
Write the game ready format with efficiency in mind
Memory, performanceSlide44
44
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
How to optimize the compilers
Write compilers with parallelism in mind
No “global variables” such as reused temp files on disc
Disc cache thrashingIt comes down to the access patternsDo as few reads/writes as possible
Use tools like Process Explorer or
vmstatSlide45
45
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
How to optimize the compilers
Do reads and writes in memory
Lowers the disc usage
E.g allocate large buffer and write into itAvoid temp files altogether
import cStringIO
memfile = cStringIO.StringIO()
MultipleWritesToAFile( mydata, memfile)
f = open(options.output, "wb" )
f.write( memfile.getvalue() )
f.close()
memfile.close()Slide46
46
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
Dare to do stuff!
If the compiler code is unreadable/inefficient
Will it be faster/more productive to rewrite it?
Rewriting will probably introduce new bugsBut you will also…Probably throw away many old bugs as well
Increase your understanding of the work flow
Get a chance to improve design/performance more directlySlide47
47
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
Dare to do stuff!
Rewriting doesn’t mean you throw the code away
You will use the code as a blue print of what, or what not, to do
It’s easy to rewrite something that does the same thing, only better!Slide48
48
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
Separate code
Use one code base to read/write data in compilers
Use another code base for loading the data in game
Keeps down code dependenciesKeeps the compilers small and simpleLess code to maintain in the compiler
Decreased code size down to 1/10
th
Less code to compile, faster turnaround times
Suitable for file formats that rarely changeSlide49
49
The Asset pipeline for Just Cause 2: Lessons Learned
Take Aways
Keep the build working!
Only replace old code with new working code
Down time is both costly and stressful
Keep it simple!Slide50
50
The Asset pipeline for Just Cause 2: Lessons Learned
Links
Python –
http://www.python.org/
waf –
http://code.google.com/p/waf wine – http://winehq.org jqplot – http://www.jqplot.com/
protobuf –
http://code.google.com/p/protobuf/
Download slides and source code on our site!Slide51
51
The Asset pipeline for Just Cause 2: Lessons Learned
Q & A
www.avalanchestudios.se/presentations
gdc@avalanchestudios.
se
Download slides and source code on our site!Slide52
52
The Asset pipeline for Just Cause 2: Lessons Learned