Re: NetBSD 5.1 TCP performance issue (lots of ACK)

[Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost>]

On 22 Nov, 2011, at 12:53 , Manuel Bouyer wrote:

> On Tue, Nov 22, 2011 at 12:23:06PM -0800, Dennis Ferguson wrote:
>> [...] 
>> Needless to say, the last bit would be entirely inconsistent with section
>> 7.2 of any version of the "Intel 64 and IA-32 Architectures Software 
>> Developer?s
>> Manual, Volume 3A: System Programming Guide, Part 1" published more recently
>> than 2007.  I won't repeat what it says here, but it is rather unambiguous
>> about the fact that newer reads (in program order) are always done after 
>> older
>> reads, at least in the basic instruction set.
> 
> I have "IA-32 Intel® Architecture Software Developer's Manual Volume 3:
> System Programming Guide" from 2004, and it says at the beggining
> of 7.2.2:
> | In a single-processor system for memory regions defined as write-back
> | cacheable, the following ordering rules apply:
> | 1. Reads can be carried out speculatively and in any order.

You are assuming the above somehow applied to Intel CPUs which existed
in 2004, but that assumption is incorrect.  There were no Intel (or AMD)
CPUs which worked like that in 2004, since post-2007 manuals document the
ordering behavior of all x86 models from the 386 forward, and explicitly
says that none of them have reordered reads, so the above could only a
statement of what they expected future CPUs might do and not what
they actually did.

This is clear in the post-2007 revision I have, where the section you quote
above now says:

  7.2.2 Memory Ordering in P6 and More Recent Processor Families

  The Intel Core 2 Duo, Intel Core Duo, Pentium 4, and P6 family processors also
  use a processor-ordered memory-ordering model that can be further defined as
  “write ordered with store-buffer forwarding.” This model can be characterized 
as follows.

  In a single-processor system for memory regions defined as write-back 
cacheable,
  the following ordering principles apply […]:

        • Reads are not reordered with other reads.
        • Writes are not reordered with older reads.

and, about speculative reads in particular, later says:

  The processor-ordering model described in this section is virtually identical 
to
  that used by the Pentium and Intel486 processors. The only enhancements in 
the Pentium 4,
  Intel Xeon, and P6 family processors are:

        • Added support for speculative reads, while still adhering to the 
ordering principles above.
        • Store-buffer forwarding, when a read passes a write to the same 
memory location.

That is, they've tightened up the guarantees for the modern processors, while 
the
older processors were in fact even more strictly ordered than this.  Speculative
reads can still occur on the modern processors, but they are now guaranteed
to be implemented in a way which still observes program ordering.  There are no
Intel processors, past or present, which work some other way.

> I can't see anything about read ordering being stronger in a SMP system.

> 
> So we have to assume that reads can be reorderd, unless we want
> NetBSD to run only on x86 systems newer than 2007 (and a lfence instruction
> is enough to enforce read ordering).
> and yes, my test system is older than 2007.

You are reading about what they thought, in 2004, that they might build in 
future.
The post-2007 manuals make clear how all processors have actually worked, from
the 386 to the modern ones, and none of them have worked that way.  If you
don't want to take my word for it then take a look at the x86 section of
this (very good) 2007 paper:

    
http://www.rdrop.com/users/paulmck/scalability/paper/ordering.2007.09.19a.pdf

which says about the same thing.  If you are using an Intel or AMD CPU (I think
Cyrix x86's might have had an out-of-order option) it won't be reordering
reads in the basic instruction set.

Dennis Ferguson