Plugins/Profiler

The Profiler is a plugin created by the torch team. It adds tools for exploring causes of lag and is also useful to check real condition mod performance. it can be downloaded here

You can run a profiling command from the server in the chat tab or as admin in the in-game chat.

The command syntax is !profile type --option1=xyz --option2=xzy ...

Example: !profile blocktypes --secs=20 --top=20

Profiling type

You can profile different parts of the game for different purposes. These types are:

frames	Gives a profile overview of the entire game
session	Gives a profile overview of the session
entities	profiles things running in MySector
factions	profiles all factions of the server
players	profiles players, meaning everything owned by a player
grids	profiles the session by grids. Connected grids count as one grid
blocktypes	Profiles all blocks in the session by type
blocks	profiles all blocks in the session
scripts	profiles all running programmable blocks
physics	profiles all havok worlds
cleangps	cleans the GPS points set by profiling

Profiling options

Most types can be run with optional commands. These change how the results get displayed, or how long the profiling will take

--secs=N	to profile N seconds (default 10sec)
--top=N	to show N entries from the top (default 10)
--player="PlayerName"	to report values for a single player
--faction="FactionTag"	to report values for a single faction
--grid="GridName"	to report values for a specific grid
--this	to profile the entity you're currently controlling (players only)
--gps	to show positional results as GPS points (players only)

Interpreting the result

Using the right commands, the profiler provides a quick way to identify problems in your world and quantifies them

The profiler will return the given most performance costing entries, by default 10.

Every entry will have a name and two numbers attached to it. These numbers are per tick, which is 16.66 ms at simspeed 1.0

This means that as soon as your world consumes more than 16.66ms computing time per tick, simspeed will drop.

The two numbers the profiler returns are The time (In ms = milliseconds/μs = microseconds/ns = nanoseconds) the entry took on the main thread and the amount of time this entry was updated in the tick (In upt = updates).

Walk-Through

so our server is laggy and we want to know why...

Make sure the server is actually laggy:

!profile sim

Best sim: 0.9
Worst sim: 0.7
Average sim: 0.8

"Best sim" came out lower than 1.0 so the server is in fact lagging.

Let's get the general idea of what's going on:

!profile frames

Session: 17.17ms/f (total 4821.84ms)
Network: 0.09ms/f (total 53.87ms)
Replication: 0.01ms/f (total 5.04ms)
GPS: 0.00ms/f (total 0.68ms)
Idle: 0.00ms/f (total 0.00ms)

Game is supposed to spend no more than 16.666ms per frame but the session is eating up 17ms.

Let's go deeper and profile the session:

!profile session

MySector: 13.66ms/f (total 4551.66ms)
MyPhysics: 2.97ms/f (total 3321.69ms)
MyProceduralWorldGenerator: 0.14ms/f (total 80.77ms)
ProductionManagerCore: 0.03ms/f (total 15.85ms)
MyGamePruningStructure: 0.02ms/f (total 10.90ms)
RelativeTopSpeed: 0.01ms/f (total 8.41ms)
MySectorWeatherComponent: 0.01ms/f (total 8.16ms)
MyPlanetEnvironmentSessionComponent: 0.01ms/f (total 7.96ms)
MySessionComponentSmartUpdater: 0.01ms/f (total 4.91ms)

There's an obvious elephant in the room but let's actually take a look at MyPhysics first:

!profile physics

0: 3 entities in 23.0km: 2.46ms/f (total 14.63ms)
1: 63 entities in 84.3km: 0.54ms/f (total 5.44ms)
2: 37 entities in 30.4km: 0.22ms/f (total 2.16ms)
3: 20 entities in 12.1km: 0.15ms/f (total 1.55ms)
4: 5 entities in 0.0km: 0.14ms/f (total 1.39ms)
5: 6 entities in 0.0km: 0.12ms/f (total 1.21ms)
6: 5 entities in 0.0km: 0.10ms/f (total 1.02ms)
7: 19 entities in 27.5km: 0.09ms/f (total 0.89ms)
8: 11 entities in 13.6km: 0.07ms/f (total 0.70ms)
9: 5 entities in 0.0km: 0.07ms/f (total 0.67ms)

Something is at 2.46ms which is unusual. Follow up with a couple more commands:

1. !profile physics --inspect=0 to list the name of grids in the laggiest physics world,
2. !profile physics --takeme=0 to show GPS of these grids on your screen,
3. !profile physics --takeme=done to hide those GPS when you're done.

We found a piston tower stuck underground and won an easy 2ms off of the session.

Let's go back and dive deeper into MySector:

!profile entities

MyCubeBlock: 9.45ms/f (total 1710.17ms)
MyCubeGrid: 3.06ms/f (total 690.59ms)
Sandbox.Game.EntityComponents/MyIngameScriptComponent: 0.83ms/f (total 497.95ms)
MyCharacter: 0.50ms/f (total 436.38ms)
MySafeZone: 0.40ms/f (total 238.91ms)
MyVoxelMap: 0.27ms/f (total 160.10ms)
MyPlanet: 0.04ms/f (total 23.92ms)
MyVoxelPhysics: 0.04ms/f (total 22.61ms)

Seems like blocks being laggy; going deeper:

!profile grids

AIS Albatross: 2.51ms/f (total 139.19ms)
Asteroid Fortress: 2.08ms/f (total 52.20ms)
AIS_Ezekiel_401: 1.78ms/f (total 51.53ms)
FSH FishToasts Dopey Nyctitropism: 1.28ms/f (total 49.20ms)
Yama: 1.08ms/f (total 48.45ms)
Alpha Base: 1.07ms/f (total 48.23ms)
TwoSnake Base 1: 0.47ms/f (total 44.67ms)
UBA. Earth Defender HQ: 0.46ms/f (total 38.64ms)
NewDawn Mk2: 0.35ms/f (total 34.34ms)

We can see several grids standing out.

Let's see what type of blocks are to blame:

!profile blocks

HydrogenThrust: 3.60ms/f (total 57.11ms)
OxygenGenerator: 2.10ms/f (total 57.16ms)
GatlingTurret: 1.18ms/f (total 110.20ms)
NpcInteriorTurret: 0.07ms/f (total 43.64ms)
CryoChamber: 0.18ms/f (total 138.77ms)
MediumCalibreTurret: 0.12ms/f (total 72.98ms)
LargeCalibreTurret: 0.10ms/f (total 60.18ms)
OxygenTank: 0.07ms/f (total 43.56ms)
CockpitOpen: 0.05ms/f (total 38.57ms)

So we've come to see that H2/gas is the major culprit in our server.

With that in mind, we can take the next steps:

1. Ask the Torch community the general number to aim for
2. Limit the number of gas-related blocks and conveyor blocks per grid
3. etc...

All these numbers will help you address and fix various performance issues in your game server. Enjoy!