As most of you probably know, Mark Probst (schani) has been hard at work in the past year on implementing a new, generational, garbage collector for Mono (called sgen). The hopes were high for this work as the currently default garbage collector in Mono, boehm, is not really up to the task and causes major slowdowns under certain workloads. The hopes were high, but Mark managed to meet and exceed them :) The test page can be found here. Below are results of a simple test, to run several thousands of requests against a small ASP.NET page, served by XSP. The first run is a "warm-up" one - that is, at the very first request gmcs is invoked in order to compile the .aspx into an assembly. The second request is "pure", without the compilation toll. The test consisted of running the following command line on the client machine (2.2GHz dual core):
ab2 -n 50000 -c 20 -k http://192.168.1.2/hi.aspx
The server was a quad 2.4GHz Xeon machine with hyper-threading on (giving 4 real and 4 "virtual" cores). The tests were conducted with Mono from the master branch. Both machines were running 64-bit Linux (OpenSuSE 11.3). The first set of results is for the boehm gc:
# ab2 -n 50000 -c 20 -k http://192.168.1.2:8080/hi.aspx
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.2 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests
Server Software: Mono.WebServer.XSP/2.7.0.0
Server Hostname: 192.168.1.2
Server Port: 8080
Document Path: /hi.aspx
Document Length: 699 bytes
Concurrency Level: 20
Time taken for tests: 34.739 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 49520
Total transferred: 51478420 bytes
HTML transferred: 34950000 bytes
Requests per second: 1439.30 [#/sec] (mean)
Time per request: 13.896 [ms] (mean)
Time per request: 0.695 [ms] (mean, across all concurrent requests)
Transfer rate: 1447.13 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 17
Processing: 4 14 26.1 10 1220
Waiting: 4 14 26.1 10 1220
Total: 4 14 26.2 10 1224
Percentage of the requests served within a certain time (ms)
50% 10
66% 11
75% 12
80% 13
90% 22
95% 38
98% 52
99% 57
100% 1224 (longest request)
# ab2 -n 50000 -c 20 -k http://192.168.1.2:8080/hi.aspx
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.2 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests
Server Software: Mono.WebServer.XSP/2.7.0.0
Server Hostname: 192.168.1.2
Server Port: 8080
Document Path: /hi.aspx
Document Length: 699 bytes
Concurrency Level: 20
Time taken for tests: 55.495 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 49518
Total transferred: 51478320 bytes
HTML transferred: 34950000 bytes
Requests per second: 900.98 [#/sec] (mean)
Time per request: 22.198 [ms] (mean)
Time per request: 1.110 [ms] (mean, across all concurrent requests)
Transfer rate: 905.88 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 13.7 0 3053
Processing: 2 22 38.9 11 1535
Waiting: 2 22 38.7 11 1535
Total: 2 22 41.3 11 3071
Percentage of the requests served within a certain time (ms)
50% 11
66% 12
75% 12
80% 13
90% 65
95% 92
98% 115
99% 241
100% 3071 (longest request)
The second set is for the sgen gc:
# ab2 -n 50000 -c 20 -k http://192.168.1.2:8080/hi.aspx
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.2 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests
Server Software: Mono.WebServer.XSP/2.7.0.0
Server Hostname: 192.168.1.2
Server Port: 8080
Document Path: /hi.aspx
Document Length: 699 bytes
Concurrency Level: 20
Time taken for tests: 28.736 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 49520
Total transferred: 51478420 bytes
HTML transferred: 34950000 bytes
Requests per second: 1739.99 [#/sec] (mean)
Time per request: 11.494 [ms] (mean)
Time per request: 0.575 [ms] (mean, across all concurrent requests)
Transfer rate: 1749.45 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 29
Processing: 4 11 27.6 10 1383
Waiting: 4 11 27.6 10 1383
Total: 4 11 27.8 10 1387
Percentage of the requests served within a certain time (ms)
50% 10
66% 11
75% 12
80% 12
90% 14
95% 15
98% 18
99% 23
100% 1387 (longest request)
# ab2 -n 50000 -c 20 -k http://192.168.1.2:8080/hi.aspx
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.2 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests
Server Software: Mono.WebServer.XSP/2.7.0.0
Server Hostname: 192.168.1.2
Server Port: 8080
Document Path: /hi.aspx
Document Length: 699 bytes
Concurrency Level: 20
Time taken for tests: 33.681 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 49512
Total transferred: 51478045 bytes
HTML transferred: 34950000 bytes
Requests per second: 1484.52 [#/sec] (mean)
Time per request: 13.472 [ms] (mean)
Time per request: 0.674 [ms] (mean, across all concurrent requests)
Transfer rate: 1492.58 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 23.6 0 3089
Processing: 2 13 24.2 10 674
Waiting: 2 13 24.2 10 674
Total: 2 13 33.8 10 3099
Percentage of the requests served within a certain time (ms)
50% 10
66% 11
75% 12
80% 13
90% 14
95% 16
98% 21
99% 172
100% 3099 (longest request)
The points of interest are:
- Requests per second. Both tests show that the second run is slower, which is to be expected since there are more objects to be collected. Boehm has much more work here since it has to perform a lot of big collections, while sgen wins by having two kinds of collections - major and minor. Sgen wins hands down here.
- Percentage of requests run in given time. Boehm shows a pretty big difference in times here, while sgen is more uniform and, once again, faster.
- Throughput. And for the third time, boehm loses here considerably - sgen by taking advantage of its minor collection feature spends less time freeing memory and therefore gives better transfer.
- Memory usage. This is not shown above, but sgen left xsp with more memory at the end of the test run than boehm (~900mb vs ~800mb), but this is expected to change as sgen development progresses.
One thing to note is that you need to run mono with the
-O=-aot option when testing sgen (at least on x86-64) since AOT somehow causes it to crash XSP.
So, two conclusions here. Sgen is going to make Mono much, much faster and, as
Gonzalo> put it (hitting the bull's eye):
<gonzalo> YOU ROCK :D (in response to Schani's question if Gonzalo was kidding regarding the performance improvement)
Subscribe