WikiXRay/Performance

From Meta, a Wikimedia project coordination wiki
(Redirected from WikiXRay Performance)

These are some performance results that give you some hints about the total time required for decompressing and parsing the 7zip dump. Don't forget that the total time you need for these tasks heavily depends on:

  • 1. The hard disk speed and CPU performance of your system. The following results has been obtained in a server platform, with 2 CPUs AMD Opteron 2212 HE Santa Rosa 2GHz (dual-core), and a RAID 6 array of 8 SATA-300 hard disks (16 KB caché, 8 msec. average seek time).
  • 2. The size of the dump you want to process (obviously). Don't forget to use 7zip dumps, because 7zip have a multithreaded implementation of the decompression process, and it saves a lot of disk space.

As you can see, the performance difference between processing a compressed 7zip file (and decompress it on the fly, as mentioned above) and processing a previously decompressed 7zip file is minor. So if you have a multi-core CPU, feel confident that the 7zip decompression will not slow down the whole process.

Allocate about 25-75%% of the following times for loading the .sql files back into MySQL (it is somewhat faster).

PTWIKI (Uncompressed XML 2007/06/21).
===========================================================================
page 964689 (129.592826 pags. per sec.), revision 6010000 (807.361634 revs. per sec.)
page 965370 (129.684310 pags. per sec.), revision 6011000 (807.495970 revs. per sec.)
page 965915 (129.740094 pags. per sec.), revision 6012000 (807.521827 revs. per sec.)
page 966548 (129.807682 pags. per sec.), revision 6013000 (807.547677 revs. per sec.)
page 967190 (129.893903 pags. per sec.), revision 6014000 (807.681977 revs. per sec.)


File successfully parsed...
page 967260 (129.903304 pags./sec.), revision 6014100 (807.695407 revs./sec.)

real    124m6.868s
user    115m7.000s
sys     3m58.470s
****************************************************************************

ITWIKI (Compressed 7zip file, decompressed on the fly and piped to dump_sax 2007/06/24)
============================================================================
page 957355 (69.398695 pags. per sec.), revision 8683000 (629.430953 revs. per sec.)
page 957617 (69.412656 pags. per sec.), revision 8684000 (629.457814 revs. per sec.)
page 957903 (69.428354 pags. per sec.), revision 8685000 (629.484671 revs. per sec.)
page 958308 (69.452674 pags. per sec.), revision 8686000 (629.511523 revs. per sec.)
page 958730 (69.483258 pags. per sec.), revision 8687000 (629.583998 revs. per sec.)
page 959385 (69.525690 pags. per sec.), revision 8688000 (629.610841 revs. per sec.)


Everything is Ok


File successfully parsed...
page 959625 (69.543083 pags./sec.), revision 8688465 (629.644539 revs./sec.)

real    229m59.953s
user    220m24.020s
sys     7m18.380s

****************************************************************************

NLWIKI (Uncompressed XML 2007/07/01)
============================================================================

page 713399 (66.000463 pags. per sec.), revision 7983000 (738.551207 revs. per sec.)
page 713830 (66.040337 pags. per sec.), revision 7984000 (738.643723 revs. per sec.)
page 714318 (66.079371 pags. per sec.), revision 7985000 (738.667900 revs. per sec.)
page 714739 (66.112201 pags. per sec.), revision 7986000 (738.692073 revs. per sec.)
page 715179 (66.152900 pags. per sec.), revision 7987000 (738.784571 revs. per sec.)
page 715646 (66.189974 pags. per sec.), revision 7988000 (738.808731 revs. per sec.)
page 716097 (66.231687 pags. per sec.), revision 7989000 (738.901221 revs. per sec.)


File successfully parsed...
page 716587 (66.270878 pags./sec.), revision 7989908 (738.916859 revs./sec.)

real    180m13.890s
user    164m10.300s
sys     5m9.410s

****************************************************************************
ESWIKI (Compressed 7z 2007/06/09)
============================================================================

page 633441 (50.213317 pags. per sec.), revision 8500000 (673.801031 revs. per sec.)
page 633839 (50.244867 pags. per sec.), revision 8501000 (673.880301 revs. per sec.)
page 634233 (50.272115 pags. per sec.), revision 8502000 (673.906151 revs. per sec.)
page 634679 (50.303479 pags. per sec.), revision 8503000 (673.931997 revs. per sec.)
page 635117 (50.338194 pags. per sec.), revision 8504000 (674.011255 revs. per sec.)
page 635635 (50.375258 pags. per sec.), revision 8505000 (674.037090 revs. per sec.)


File successfully parsed...
page 636134 (50.410809 pags./sec.), revision 8505915 (674.056185 revs./sec.)
*****************************************************************************

SVWIKI (Compressed 7zip 2007/07/06)
=============================================================================

page 510694 (128.832997 pags. per sec.), revision 4350000 (1097.376387 revs. per sec.)
page 511149 (128.915259 pags. per sec.), revision 4351000 (1097.351828 revs. per sec.)
page 511636 (129.038083 pags. per sec.), revision 4352000 (1097.604035 revs. per sec.)


Everything is Ok


File successfully parsed...
page 511747 (129.033535 pags./sec.), revision 4352233 (1097.386031 revs./sec.)

real    66m6.912s
user    61m59.220s
sys     2m22.740s
*****************************************************************************

PLWIKI (Compressed 7zip 2007/06/29)
=============================================================================
page 661985 (75.371172 pags. per sec.), revision 7722000 (879.198452 revs. per sec.)
page 662365 (75.405852 pags. per sec.), revision 7723000 (879.212204 revs. per sec.)
page 662717 (75.437336 pags. per sec.), revision 7724000 (879.225953 revs. per sec.)
page 663083 (75.470407 pags. per sec.), revision 7725000 (879.239700 revs. per sec.)
page 663488 (75.516504 pags. per sec.), revision 7726000 (879.353517 revs. per sec.)
page 663982 (75.564129 pags. per sec.), revision 7727000 (879.367247 revs. per sec.)


Everything is Ok


File successfully parsed...
page 664347 (75.605667 pags./sec.), revision 7727722 (879.449414 revs./sec.)

real    146m27.817s
user    140m37.000s
sys     4m47.090s


*****************************************************************************

FRWIKI (Compressed 7zip 2007/06/12)
=============================================================================
page 1345787 (51.454292 pags. per sec.), revision 16980000 (649.206653 revs. per sec.)
page 1346259 (51.470370 pags. per sec.), revision 16981000 (649.220064 revs. per sec.)
page 1346698 (51.485186 pags. per sec.), revision 16982000 (649.233475 revs. per sec.)
page 1347213 (51.504874 pags. per sec.), revision 16983000 (649.271705 revs. per sec.)
page 1347618 (51.518388 pags. per sec.), revision 16984000 (649.285114 revs. per sec.)


Everything is Ok


File successfully parsed...
page 1347933 (51.528461 pags./sec.), revision 16984593 (649.282962 revs./sec.)

real    435m59.077s
user    423m36.670s
sys     13m5.270s


*****************************************************************************