Eric Dumazet 3d9a0d2f82 dql: dql_queued() should write first to reduce bus transactions
While doing high throughput test on a BQL enabled NIC,
I found a very high cost in ndo_start_xmit() when accessing BQL data.

It turned out the problem was caused by compiler trying to be
smart, but involving a bad MESI transaction :

  0.05 │  mov    0xc0(%rax),%edi    // LOAD dql->num_queued
  0.48 │  mov    %edx,0xc8(%rax)    // STORE dql->last_obj_cnt = count
 58.23 │  add    %edx,%edi
  0.58 │  cmp    %edi,0xc4(%rax)
  0.76 │  mov    %edi,0xc0(%rax)    // STORE dql->num_queued += count
  0.72 │  js     bd8

I got an incredible 10 % gain [1] by making sure cpu do not attempt
to get the cache line in Shared mode, but directly requests for
ownership.

New code :
	mov    %edx,0xc8(%rax)  // STORE dql->last_obj_cnt = count
	add    %edx,0xc0(%rax)  // RMW   dql->num_queued += count
	mov    0xc4(%rax),%ecx  // LOAD dql->adj_limit
	mov    0xc0(%rax),%edx  // LOAD dql->num_queued
	cmp    %edx,%ecx

The TX completion was running from another cpu, with high interrupts
rate.

Note that I am using barrier() as a soft hint, as mb() here could be
too heavy cost.

[1] This was a netperf TCP_STREAM with TSO disabled, but GSO enabled.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 00:04:55 -04:00
..
2014-05-06 17:54:53 -04:00
2014-08-07 17:17:39 -07:00
2014-08-13 18:13:19 -06:00
2014-09-26 15:05:15 -04:00
2014-08-01 22:35:55 +08:00
2014-08-05 15:59:35 -07:00
2014-07-08 13:36:52 -07:00
2014-06-26 15:46:55 -03:00
2014-08-22 13:56:47 -07:00
2014-04-01 23:19:14 -04:00
2014-03-10 11:44:42 -04:00
2014-09-26 15:05:15 -04:00
2014-08-07 14:40:08 -04:00
2014-04-18 14:20:48 +02:00
2014-08-06 18:01:24 -07:00
2014-05-20 10:30:40 +09:00
2014-02-20 14:54:28 +01:00
2014-08-06 18:01:19 -07:00
2014-08-19 10:19:39 -05:00
2014-06-06 16:08:13 -07:00
2014-06-02 16:05:24 -07:00
2014-06-02 14:56:01 -07:00
2014-07-08 09:40:06 +02:00
2014-04-17 12:30:40 -07:00
2014-06-16 16:26:54 +02:00
2014-03-13 12:11:00 +10:30
2014-07-22 21:55:45 +01:00
2014-05-27 17:38:11 -07:00
2014-07-17 22:07:37 +02:00
2014-03-04 13:51:06 -05:00
2014-04-01 23:19:10 -04:00
2014-04-07 10:59:19 -07:00
2014-06-10 14:35:22 -07:00
2014-07-09 14:58:37 +01:00
2014-06-04 16:54:07 -07:00
2014-06-04 16:54:21 -07:00
2014-07-15 22:40:22 +02:00
2014-06-04 16:54:12 -07:00
2014-09-13 12:30:26 -04:00
2014-08-08 15:57:26 -07:00
2014-08-08 15:57:31 -07:00
2014-08-06 13:03:44 +02:00
2014-09-26 15:40:06 -04:00
2014-06-12 00:21:11 -04:00
2014-02-17 15:01:37 -08:00
2014-08-08 15:57:17 -07:00
2014-06-04 16:53:57 -07:00
2014-08-08 15:57:24 -07:00
2014-08-22 21:28:02 -07:00
2014-06-04 16:53:56 -07:00
2014-05-14 09:46:32 -07:00
2014-08-08 15:57:22 -07:00
2014-09-16 15:02:55 -06:00
2014-06-04 16:53:57 -07:00
2014-04-07 16:35:53 -07:00
2014-06-04 16:53:57 -07:00