Skip to content

Performance Tests

Jump to bottom
ricardoquesada edited this page Dec 3, 2014 · 8 revisions

How to run performance tests

Basic rules

  • Never run the tests on a Simulator and/or Emulator. Always use real devices
  • The older the device the better
  • Never run the tests in DEBUG mode. Always use RELEASE mode
  • Double check that RELEASE mode is using all the possible optimizations for the C++ code

Devices are multitasking, and many tasks are run in the background and those tasks could affect the performance. In order to minimize that do:

  • Turn Airplane mode ON
  • Disable Wifi
  • Disable Bluetooth
  • Kill all running tasks / applications

Logging

  • Performance Tests must be run before releasing an stable version
  • The results must be logged in this spreadsheet
  • In order to know the performance of the new release, a comparison must be done with the previous versions. The comparison must try to use the same testing environment. As an example:
    • Same devices (MUST)
    • Same device operating system (SHOULD)
    • Same compiler and toolchain (DESIRABLE)

Starting from Cocos2d-x v3.3, the "auto run" feature of Sprite Performance Tests logs in the console the output of the tests. This output can be copy & pasted into the spreadsheet

Measuring the performance

Most of the Cocos2d-x performance tests relies on watching the FPS number. Although this is not a reliable way to test how fast a particular component is, it gives us at least a way to know how fast Cocos2d-x is in general.

Note: The method of taking a look at the FPS MUST NOT be used to measure the performance of a particular algorithm, or function. Instead Sampling Profiling like Xcode Instruments, or Linux Perf must be used. Intrusive Profiling is also valid method, like using the CC_PROFILER_START and CC_PROFILER_STOP macros.

Looking at the FPS manually

Pros:

  • Doesn't require additional services / tools:
    • Wifi can be turned off
    • Xcode / Visual Studio are not needed
  • Can be used to test any kind of tests:
    • eg: Sprite Test with 5250 sprites

Cons:

  • Like any other manual process, it is error prone and takes a lot of time.

Suggested for:

  • Only to test a few tests. Usually to make sure that the Automated test output is correct.

Automated Tests: cocos2d console

The automated tests (as of v3.3 only Sprite Performance Test is automated) logs the results using the cocos2d::Console::log() function. That means that the cocos2d::Console has the output. And it is possible to get the output by connecting the cocos2d::Console by using telnet:

$ telnet 10.0.1.16 5678

10.0.1.16 would the IP address of the device.

Once inside the console, you can type help to see all the available commands:

!https://lh3.googleusercontent.com/-fU0BouzMLA8/VH-IFP7vbfI/AAAAAAABJ68/gywTItn0_TY/s640/Screen%2520Shot%25202014-12-03%2520at%25202.00.35%2520PM.png

By default, cpp-tests enables the cocos2d::Console and that can be used obtain the output of the console.

  1. Automatically: Run the tests from Xcode (or Visual Studio or Instruments) and use Xcode's console to see the cocos2d output
  2. Automatically: Run the tests from the device, and connect to the cocos2d console using telnet

1 should be avoided since it is manual. 3 is preferred over 2, at least for old iOS devices. 2 affected the performance on iOS 6 on and iPod Touch 4.

Clone this wiki locally