[msaitoh%execsw.org@localhost: Re: kern/53903: ixgbe cannot access some PCI registers]

To: kern-bug-people%netbsd.org@localhost,gnats-admin%netbsd.org@localhost,netbsd-bugs%netbsd.org@localhost,maekawa%daemon-systems.org@localhost
Subject: [msaitoh%execsw.org@localhost: Re: kern/53903: ixgbe cannot access some PCI registers]
From: David Holland <dholland-bugs%netbsd.org@localhost>
Date: Fri, 25 Jan 2019 17:15:02 +0000 (UTC)
The following reply was made to PR kern/53903; it has been noted by GNATS.

From: David Holland <dholland-bugs%netbsd.org@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: [msaitoh%execsw.org@localhost: Re: kern/53903: ixgbe cannot access some PCI
 registers]
Date: Fri, 25 Jan 2019 17:12:03 +0000

 not sent to gnats
 (when replying to your own postings it's often necessary to explicitly
 set To: to gnats-bugs)
 
    ------
 
 From: Masanobu SAITOH <msaitoh%execsw.org@localhost>
 To: kern-bug-people%netbsd.org@localhost, gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost,
 	maekawa%daemon-systems.org@localhost
 Subject: Re: kern/53903: ixgbe cannot access some PCI registers
 Date: Fri, 25 Jan 2019 14:08:19 +0900
 
 On 2019/01/24 12:10, Masanobu SAITOH wrote:
 > The following reply was made to PR kern/53903; it has been noted by GNATS.
 > 
 > From: Masanobu SAITOH <msaitoh%execsw.org@localhost>
 > To: gnats-bugs%NetBSD.org@localhost, kern-bug-people%netbsd.org@localhost,
 >   gnats-admin%netbsd.org@localhost, netbsd-bugs%netbsd.org@localhost
 > Cc: msaitoh%execsw.org@localhost
 > Subject: Re: kern/53903: ixgbe cannot access some PCI registers
 > Date: Thu, 24 Jan 2019 12:07:41 +0900
 > 
 >   On 2019/01/23 23:05, maekawa%daemon-systems.org@localhost wrote:
 >   >> Number:         53903
 >   >> Category:       kern
 >   >> Synopsis:       ixgbe cannot access some PCI registers
 >   >> Confidential:   no
 >   >> Severity:       serious
 >   >> Priority:       medium
 >   >> Responsible:    kern-bug-people
 >   >> State:          open
 >   >> Class:          sw-bug
 >   >> Submitter-Id:   net
 >   >> Arrival-Date:   Wed Jan 23 14:05:01 +0000 2019
 >   >> Originator:     MAEKAWA Masahide
 >   >> Release:        8.99.31
 >   >> Organization:
 >   > M-Systems, Inc.
 >   >> Environment:
 >   > NetBSD hpegen10.daemon-systems.org 8.99.31 NetBSD 8.99.31 (GENERIC) #0: Wed Jan 23 21:59:19 JST 2019  maekawa%kerberos.daemon-systems.org@localhost:/home/release/objs/sys/arch/amd64/compile/GENERIC amd64
 >   >> Description:
 >   > X540-T1 doesn't work on MicroServer Gen10 when "UEFI only", enabling "Above 4G Decoding" in setup menu.
 >   > When disable "Above 4G Decoding", ixg(4) is attached normaly.
 >   >
 >   > Full dmesg is : https://www.daemon-systems.org/~maekawa/ixgbe/dmesg_64bit_regdump.log
 >   >
 >   > The ixg(4)-related part of boot message:
 >   > ixg0 at pci3 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 4.0.1-k
 >   > ixg0: clearing prefetchable bit
 >   > ixgbe_init_shared_code
 >   > ixgbe_set_mac_type
 >   >
 >   > ixgbe_set_mac_type found mac: 4, returns: 0
 >   >
 >   > ixgbe_init_ops_X540
 >   > ixgbe_init_phy_ops_generic
 >   > ixgbe_init_ops_generic
 >   > ixgbe_get_pcie_msix_count_generic
 >   > ixgbe_init_ops_X540 end
 >   > ixg0: device X540
 >   > ixgbe_set_lan_id_multi_port_pcie
 >   > ixgbe_get_device_caps_generic
 >   > ixgbe_read_eerd_X540
 >   > ixgbe_acquire_swfw_sync_X540
 >   > ixgbe_get_swfw_sync_semaphore
 >   > ixg0: Software semaphore SMBI between device drivers not granted.
 >   > Failed to get NVM access and register semaphore, returning IXGBE_ERR_SWFW_SYNC
 >   >
 >   > 00000: ffffffff
 >   > 00004: ffffffff
 >   > 00008: ffffffff
 >   > 0000c: ffffffff
 >   > 00010: ffffffff
 >   > 00014: ffffffff
 >   > 00018: ffffffff
 >   > 0001c: ffffffff
 >   > 00020: ffffffff
 >   > 00024: ffffffff
 >   > 00028: ffffffff
 >   > 0002c: ffffffff
 >   > 00030: ffffffff
 >   > 00034: ffffffff
 >   > 00038: ffffffff
 >   > 0003c: ffffffff
 >   > 10140: ffffffff
 >   > 10160: ffffffff
 >   > 10140: ffffffff
 >   > 10160: ffffffff
 >   The above diff was generated from the following diff:
 >   Index: ixgbe.c
 >   ===================================================================
 >   RCS file: /cvsroot/src/sys/dev/pci/ixgbe/ixgbe.c,v
 >   retrieving revision 1.173
 >   diff -u -p -r1.173 ixgbe.c
 >   --- ixgbe.c	23 Jan 2019 09:47:52 -0000	1.173
 >   +++ ixgbe.c	23 Jan 2019 11:44:40 -0000
 >   @@ -876,6 +871,23 @@ ixgbe_attach(device_t parent, device_t d
 >     	} else
 >     		adapter->num_segs = IXGBE_82598_SCATTER;
 >   +#if 1
 >   +	{
 >   +		uint32_t i;
 >   +
 >   +		for (i = 0; i < 64; i += 4)
 >   +			printf("%05x: %08x\n", i, IXGBE_READ_REG(hw, i));
 >   +		printf("%05x: %08x\n", IXGBE_SWSM_BY_MAC(hw),
 >   +		    IXGBE_READ_REG(hw, IXGBE_SWSM_BY_MAC(hw)));
 >   +		printf("%05x: %08x\n", IXGBE_SWFW_SYNC_BY_MAC(hw),
 >   +		    IXGBE_READ_REG(hw, IXGBE_SWFW_SYNC_BY_MAC(hw)));
 >   +		printf("%05x: %08x\n", IXGBE_SWSM_BY_MAC(hw),
 >   +		    IXGBE_READ_REG(hw, IXGBE_SWSM_BY_MAC(hw)));
 >   +		printf("%05x: %08x\n", IXGBE_SWFW_SYNC_BY_MAC(hw),
 >   +		    IXGBE_READ_REG(hw, IXGBE_SWFW_SYNC_BY_MAC(hw)));
 >   +	}
 >   +#endif
 >   +
 >     	/* Ensure SW/FW semaphore is free */
 >     	ixgbe_init_swfw_semaphore(hw);
 >   > ixgbe_get_swfw_sync_semaphore
 >   > ixg0: Software semaphore SMBI between device drivers not granted.
 >   > ixgbe_release_swfw_sync_semaphore
 >   > ixgbe_acquire_swfw_sync_X540
 >   > ixgbe_get_swfw_sync_semaphore
 >   > ixg0: Software semaphore SMBI between device drivers not granted.
 >   > Failed to get NVM access and register semaphore, returning IXGBE_ERR_SWFW_SYNC
 >   >
 >   > ixgbe_release_swfw_sync_X540
 >   > ixgbe_get_swfw_sync_semaphore
 >   > ixg0: Software semaphore SMBI between device drivers not granted.
 >   > ixgbe_release_swfw_sync_semaphore
 >   > ixgbe_reset_hw_X540
 >   > ixgbe_stop_adapter_generic
 >   > ixgbe_disable_pcie_master
 >   > GIO Master Disable bit didn't clear - requesting resets
 >   >
 >   > ixgbe_acquire_swfw_sync_X540
 >   > ixgbe_get_swfw_sync_semaphore
 >   > ixg0: Software semaphore SMBI between device drivers not granted.
 >   > Failed to get NVM access and register semaphore, returning IXGBE_ERR_SWFW_SYNC
 >   >
 >   > ixg0: semaphore failed with -16ixg0: autoconfiguration error: Hardware initialization failed
 >   >> How-To-Repeat:
 >   > 0.) Install X540-T1 to MicroServer Gen10
 >   > 1.) Restore default settings in setup menu.
 >   > 2.) Enable "Above 4G Decoding" in setup menu.
 >   > 3.) Boot
 >   >> Fix:
 >   >
 >   The following directory has some logs:	
 >   	https://www.daemon-systems.org/~maekawa/ixgbe/
 >   pcidump_{32,64}bit.log are dump of all pci devices
 >   (64 is from "Above 4G Decoding" enabled).
 >     Only X540 is mapped to over 4G. I suspect the reason is that
 >   the BAR0 is marked prefetchable and it's used as a annotation
 >   that the device requires large amounts of memory space. See also:
 >   	http://mail-index.netbsd.org/tech-kern/2017/03/22/msg021678.html
 >   I think this problem is not ixgbe specific.
 >   Has our implementation of amd64 PCI bus access 32bit limitation?
 >   (extent_create() or extent_alloc())
 >     I'm not familiar with x86 PCI memory map, so any test patch are
 >   welcomed :)
 
  I have SuperMicro A2SDi-H-TP4F. The BIOS setting has "Above 4G Decoding"
 enable/disable flag. I changed it to enable. The BIOS set the BAR0 over
 4GB and the driver WORKS without any problem.
 
 >     Base address register at 0x10
 >       type: 64-bit prefetchable memory
 >       base: 0x0000007d7f600000
 
 So, I think this is not ixgbe driver's problem but it might be
 BIOS/UEFI, our ACPI code, or some others' problem...
 
 -- 
 -----------------------------------------------
                 SAITOH Masanobu (msaitoh%execsw.org@localhost
                                  msaitoh%netbsd.org@localhost)
Prev by Date: Re: PR/53904 CVS commit: src/lib/libc/gen
Next by Date: Re: PR/53904 CVS commit: src/lib/libc/gen
Previous by Thread: toolchain/53909: broken types on aarch64
Next by Thread: PR/53908 CVS commit: src
Indexes:
Home | Main Index | Thread Index | Old Index