User talk:Milimetric (WMF)/Work log/2015-06-26

Event Logging load test

Setup:

• using analytics1004.eqiad.wmnet, a beefy if not trusty Cisco box
• 31492 events, 4 schemas, split 3/3/3/1, generated by the wonderful test_load.py
• format: seqId[space]?[qson event]

Processor

• 4 runs average: 17.6478 seconds, so ~1784 events per second
• command: time cat /home/milimetric/load.test.seq.30k | python ./eventlogging-processor "%{seqId}d %q" stdin:// file:///home/milimetric/out.load.30k

Consumer

• NOTE: mysql server was installed on the same host as the eventlogging-consumer
• NOTE: tried this with four different batch sizes (400, 1000, 4000, 8000) and it didn't seem to make a substantive difference. Therefore I just used the default configured batch size of 400. This indicates that most likely the performance bottleneck is in preparing and organizing the events for insertion and not the insertion itself.
• 4 runs average: 11.8945 seconds, so ~2647 events per second
• command: time cat ~/out.load.30k | python eventlogging-consumer stdin:// mysql://root:root@127.0.0.1/log?charset=utf8