-
Notifications
You must be signed in to change notification settings - Fork 7.1k
Performance Tests
- Never run the tests on a Simulator and/or Emulator. Always use real devices
- The older the device the better
- Never run the tests in DEBUG mode. Always use RELEASE mode
- Double check that RELEASE mode is using all the possible optimizations for the C++ code
Devices are multitasking, and many tasks are run in the background and those tasks could affect the performance. In order to minimize that do:
- Turn Airplane mode ON
- Disable Wifi
- Disable Bluetooth
- Kill all running tasks / applications
- Performance Tests must be run before releasing an stable version
- The results must be logged in this spreadsheet
- In order to know the performance of the new release, a comparison must be done with the
previous versions. The comparison must try to use the same testing environment. As an example:
- Same devices (MUST)
- Same device operating system (SHOULD)
- Same compiler and toolchain (DESIRABLE)
Starting from Cocos2d-x v3.3, the "auto run" feature of Sprite Performance Tests logs in the console the output of the tests. This output can be copy & pasted into the spreadsheet
Most of the Cocos2d-x performance tests relies on watching the FPS number. Although this is not a reliable way to test how fast a particular component is, it gives us at least a way to know how fast Cocos2d-x is in general.
Note: The method of taking a look at the FPS MUST NOT be used to measure the performance of a particular algorithm, or function.
Instead Sampling Profiling like Xcode Instruments, or Linux Perf must be used. Intrusive Profiling is also valid method, like using the CC_PROFILER_START
and CC_PROFILER_STOP
macros.
Pros:
- Doesn't require additional services / tools:
- Wifi can be turned off
- Xcode / Visual Studio are not needed
- Can be used to test any kind of tests:
- eg: Sprite Test with 5250 sprites
Cons:
- Like any other manual process, it is error prone and takes a lot of time.
Suggested for:
- Only to test a few tests. Usually to make sure that the Automated test output is correct.
The automated tests (as of v3.3 only Sprite Performance Test is automated) logs the results using the cocos2d::Console::log()
function. That means that the cocos2d::Console
has the output. And it is possible to get the output by connecting the cocos2d::Console
by using telnet:
$ telnet 10.0.1.16 5678
10.0.1.16 would the IP address of the device.
Once inside the console, you can type help to see all the available commands:
![https://lh3.googleusercontent.com/-fU0BouzMLA8/VH-IFP7vbfI/AAAAAAABJ68/gywTItn0_TY/s640/Screen%2520Shot%25202014-12-03%2520at%25202.00.35%2520PM.png]
By default, cpp-tests enables the cocos2d::Console
and that can be used obtain the output of the console.
- Automatically: Run the tests from Xcode (or Visual Studio or Instruments) and use Xcode's console to see the cocos2d output
- Automatically: Run the tests from the device, and connect to the cocos2d console using telnet
1 should be avoided since it is manual. 3 is preferred over 2, at least for old iOS devices. 2 affected the performance on iOS 6 on and iPod Touch 4.