Re: Xen kernel broken for me

["Ben C." <armondbc%gmail.com@localhost>]

Yes, - I migrated to a different box and forgot to change the console=
line  (was set to com0, needed to be pc to get output directed
properly).  After I switched it yesterday, I'm getting output very
similar to yours with newer kernels (unfortunately I have to use VNC
which isn't copy&paste capable, the best I could provide is a png
screenshot without actually retyping it)

I'm 100% sure the setup will work with the uncompressed kernel (if you
notice, it's unpacked with gunzip) or at least it always has.  I was
under the impression that it /wouldn't/ work gzipped, but I guess that
doesn't matter.

I would also like to clarify, it looks like the kernel from source
6/15 and before COULD of worked with the xen3 hypervisor.  I had a
typo in my /boot.cfg from the point I was testing sources from 6/15
and "backwards" so it's very possible some or all of these could of
worked, but my typo in /boot.cfg made it fail.

On that note I did some re-testing late last night, and I am certain
at least the latest -CURRENT is broken with the xen3 hypervisor.

I will get some more tests later today to try to track down the issue
and I will also test the xen 4.1 hypervisor to see if it works.  If
xen4 works, I don't have much of a problem, but somebody should really
let somebody know!

Thanks, Ben Carroll

On Fri, Jul 15, 2011 at 5:46 AM, Roger Pau Monné
<roger.pau%entel.upc.edu@localhost> wrote:
> Hello,
>
> I use Xen 4.1.1 right now, and I can confirm it works with XEN3_DOM0
> (amd64) kernel 2011-06-09 01:30 UTC. The line I use to boot is:
>
> menu=NetBSD/XEN41(serial):load /netbsd.xen console=com0;multiboot
> /xen.gz dom0_mem=512M dom0_max_vcpus=1 dom0_vcpus_pin console=com1
> com1=115200,8n1
>
> I see that you use /xen instead of /xen.gz, are you sure this is
> correct? If you don't see any output this means that not even your xen
> kernel is booting, even if you have a bad netbsd kernel, you should at
> least see the output of the xen kernel booting (xen kernel boots
> before netbsd kernel).
>
> Regards, Roger.
>
> 2011/7/15 Ben C. <armondbc%gmail.com@localhost>:
>> It's quite possible.  I don't get any output, while it seems you get a
>> plethora of output.
>>
>> Secondly, I actually tried backing it up ALL the way to 11/30/2010 and
>> still didn't get a boot which I find very odd.  Although when running
>> -current source over 6 months old, it's hard for me to complain about
>> that aspect.
>>
>> Rodger, can you please procide me with the line you are using to boot
>> in /boot.cfg and also if you are using the xen or xen-debug kernel and
>> I'll see if I can't get any output?  Also, have you tried the xen4
>> hypervisor kernel?
>>
>> Just for ref., here is almost line-by-line how I'm getting to the
>> point of non-booting, straight from my recipe book:
>>
>> # Install pkgsrc ---------------------------------
>> cd /usr
>> ftp ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz
>> tar xfvz pkgsrc.tar.gz
>> rm pkgsrc.tar.gz
>>
>> # Install vim ----------------------------------
>> echo ACCEPTABLE_LICENSES= vim-license >> /etc/pkg_install.conf
>> cd /usr/pkgsrc/editors/vim/
>> make install clean
>>
>> # Install src for custom kernel -------------------------------------------
>> mkdir /root/.ssh
>> echo 'anoncvs.us.netbsd.org,204.152.190.16 ssh-rsa
>> AAAAB3NzaC1yc2EAAAABIwAAAIEA3QiBl8leG9fqIJpKeNov0PKq5YryFFiroMWOPUv4hDFn8R0jC07YVaR/OSBrr37CTmGX5AFceXPzoFnLlwCqWR7rXg4NR75FTlTp9CG9EBAEtU8mee27KDrUFBTZdfVl2+aRYoAI5fTXA+0vpIO68Cq843vRWUZCcwinS4cNLUU='
>>> /root/.ssh/known_hosts
>> cd /usr
>> # this is how I was pulling old src's for example
>> #cvs -z3 -d anoncvs%anoncvs.us.netbsd.org@localhost:/cvsroot co -P -D
>> '2011-05-30 00:00:00' src
>> # but this gets current..
>> cvs -z3 -d anoncvs%anoncvs.us.netbsd.org@localhost:/cvsroot co -P src
>>
>> mkdir /usr/mybuild /usr/mybuild/obj /usr/mybuild/tools
>> /usr/mybuild/dest /usr/mybuild/release /usr/mybuild/xsrc
>>
>> # Never got far enough to make this useful, but it was indeed present
>> when building my kernels
>> echo MKLVM=yes >> /etc/mk.conf
>>
>>
>> # netbsd_t was my "Development Generic Kernel"
>> # to compare against the xen kernels to see if it was an overall
>> netbsd issue or specifically related to xen,
>> # which from what I can tell it is indeed very much related to xen
>>
>> cd /usr/src
>> ./build.sh -T /usr/mybuild/tools -D /usr/mybuild/dest -R
>> /usr/mybuild/release -O /usr/mybuild/obj -X /usr/mybuild/xsrc -j8
>> tools kernel=GENERIC
>> cp /netbsd_t /netbsd_t.`date +%F.%s`
>> cp /usr/mybuild/obj/sys/arch/amd64/compile/GENERIC/netbsd /netbsd_t
>>
>> # I also have other parts of the receipe that included a kernel called
>> XEN3_DOM0-SPEED which removed debugging and some other things,
>> # but this is indeed what was tested, never got as far as to trying to
>> refine the kernel:
>>
>> cp /usr/src/sys/arch/amd64/conf/XEN3_DOM0
>> /usr/src/sys/arch/amd64/conf/XEN3_DOM0-GENERIC
>> cd /usr/src
>> ./build.sh -T /usr/mybuild/tools -D /usr/mybuild/dest -R
>> /usr/mybuild/release -O /usr/mybuild/obj -X /usr/mybuild/xsrc -j8 -r
>> tools kernel=XEN3_DOM0-GENERIC
>> cp /XEN3_DOM0-GENERIC /XEN3_DOM0-GENERIC.`date +%F.%s`
>> cp /usr/mybuild/obj/sys/arch/amd64/compile/XEN3_DOM0-GENERIC/netbsd
>> /XEN3_DOM0-GENERIC
>>
>> # Install xentools33 and xenkernel33 (hypervisor kernel) ------------------
>> cd /usr/pkgsrc/sysutils/xentools33
>> make install clean
>> cd /usr/pkgsrc/sysutils/xenkernel33
>> make install clean
>> cp /usr/pkg/xen3-kernel/xen.gz /
>> cp /usr/pkg/xen3-kernel/xen-debug.gz /
>> gunzip /xen.gz
>> gunzip /xen-debug.gz
>>
>> # Pretty much the rest except boot.cfg, is probably not relevant, but
>> included here for completeness:
>>
>> cd /dev
>> ./MAKEDEV xen
>> sh MAKEDEV vnd4 vnd5 vnd6 vnd7 vnd8 vnd9 vnd10 vnd11 vnd12 vnd13 vnd14
>> vnd15 vnd16 vnd17 vnd18 vnd19 vnd20 vnd21 vnd22 vnd23 vnd24 vnd25
>> vnd26 vnd27 vnd28 vnd29 vnd30 vnd31 vnd32 vnd33 vnd34 vnd35 vnd36
>> vnd37 vnd38 vnd39 vnd40 vnd41 vnd42 vnd43 vnd44 vnd45 vnd46 vnd47
>> vnd48 vnd49 vnd50 vnd51 vnd52 vnd53 vnd54 vnd55 vnd56 vnd57 vnd58
>> vnd59 vnd60 vnd61 vnd62 vnd63 vnd64 vnd65 vnd66 vnd67 vnd68 vnd69
>> vnd70 vnd71 vnd72 vnd73 vnd74 vnd75 vnd76 vnd77 vnd78 vnd79 vnd80
>> vnd81 vnd82 vnd83 vnd84 vnd85 vnd86 vnd87 vnd88 vnd89 vnd90 vnd91
>> vnd92 vnd93 vnd94 vnd95 vnd96 vnd97 vnd98 vnd99 vnd100 vnd101 vnd102
>> vnd103 vnd104 vnd105 vnd106 vnd107 vnd108 vnd109 vnd110 vnd111 vnd112
>> vnd113 vnd114 vnd115 vnd116 vnd117
>>
>> echo 'sshd=YES
>> xend=YES
>> xenbackendd=YES
>> xendomains=""
>> ' >> /etc/rc.conf
>>
>> echo 'menu=Boot normally:boot netbsd
>> menu=Boot single user:boot netbsd -s
>> menu=Disable ACPI:boot netbsd -2
>> menu=Drop to boot prompt:prompt
>> menu=Development GENERIC kernel:boot netbsd_t
>> menu=Boot Xen with 768MB GENERIC XEN and inc. loglvl:load
>> /XEN3_DOM0-GENERIC console=com0;multiboot /xen dom0_mem=768M
>> loglvl=all guest_loglvl=all com1=9600,8n1 console=com1
>> menu=Boot Xen-debug with 768MB GENERIC XEN and inc. loglvl:load
>> /XEN3_DOM0-GENERIC console=com0;multiboot /xen-debug dom0_mem=768M
>> loglvl=all guest_loglvl=all com1=9600,8n1 console=com1
>> default=6
>> timeout=8
>> clear=1
>> ' > /boot.cfg
>>
>> reboot
>>
>>
>> ----
>>
>> I would also note a few modifications:
>> 1) I tried shortening the filename of the kernel at one point.  I
>> don't think netbsd would have a problem even with extremely long
>> kernel names, but just in case, it was tried on /some/ of the kernels
>> I compiled throughout the last few days.
>>
>> 2) At several points I tried increasing the memory from 768MB of RAM
>> to 2GB (dom0_mem=2048M)
>>
>> Thanks, Ben
>>
>> On Thu, Jul 14, 2011 at 3:06 PM, Roger Pau Monné
>> <roger.pau%entel.upc.edu@localhost> wrote:
>>> I've filled a pr the past month, some of the recently merged branches
>>> broke the dom0 xen kernel, here is the bug report:
>>>
>>> http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=45062
>>>
>>> Take a look to see if it's related.
>>>
>>> Regards, Roger.
>>>
>>> 2011/7/14 Ben C. <armondbc%gmail.com@localhost>:
>>>> Couldn't decide if it was more appropriate for ports-xen or
>>>> current-users.  I figured current-users since it's more to do with the
>>>> -current branch than xen, maybe.  I'm not sure, so apologies if anyone
>>>> feels this is in the wrong place.
>>>>
>>>> on July 9th, I ran cvsup on my src tree, and rebuilt the XEN3-DOM0
>>>> kernel.  At the time I was using a slightly customized kernel
>>>> configuration based on the stock XEN3-DOM0 kernel config.  Mine just
>>>> turned off some debugging and tracing.  The server never came back up
>>>> after rebooting the server with the new kernel and, the majority of
>>>> the events with that are irrelevant at this point.  Eventually, I got
>>>> a hold of some new hardware and have been attempting to install
>>>> XEN3-DOM0 on it for about 2 days.
>>>>
>>>> I've ran cvsup's on July 9th, 10th, & 11th and essentially the
>>>>     /usr/src/sys/arch/amd64/conf/GENERIC kernel seems working on my
>>>> system, just for a comparison.  It works fine.
>>>>     /usr/src/sys/arch/amd64/conf/XEN3_DOM0-GENERIC kernel is broken.
>>>> I can't provide much debugging information.  All I can tell you is the
>>>> datacenter says "The screen shows a white underscore (_) blinking on
>>>> the screen, and nothing ever happens."
>>>>
>>>> I backed up the NetBSD src up a bit and June 25th was also showing the
>>>> same behavior.  At this point, I ditched my real hardware (I'm pretty
>>>> sure at this point my datacenter was ready to throw me to the curb
>>>> after 10-15 manual reboots and 3 reinstalls in a matter of days).  I
>>>> setup a NetBSD domu running HVM, and then tried a variety of Xen Dom0
>>>> kernels (under this HVM).  I tracked it down to something between 5/30
>>>> and 6/05 that "broke" my hardware.  I could be wrong, but this seems
>>>> to be the case.
>>>>
>>>> That is, the kernel that came with the source from 6/05, 6/10, 6/25,
>>>> 6/30, 7/9, 7/10, 7/11 and HEAD as of sometimes about 9AM today, the
>>>> 14th, is non-working for me.  They all have the same behavior as
>>>> mentioend above: GENERIC work, the XEN3-DOM0 kernel does not.  Once
>>>> again, so far all the datacenter techs can tell me is "it shows a
>>>> blinking underscore (_)" and never boots. -- this was all on the i7
>>>>
>>>> The i5 is running right now on a kernel from 5/30 or so.  I'm waiting
>>>> for the datacenter to reboot my box into the generic kernel again and
>>>> I'm going to test the i7 with the source from 5/30.
>>>>
>>>> Is anyone else experiencing issues?
>>>>
>>>> My original hardware was 2.8GHz i5 760 with 8GB of RAM.
>>>> My newer hardware is a 2.93GHz i7 870 with 16GB of RAM.
>>>>
>>>> Please excuse any ignorance encountered in the above.  Thanks,
>>>>
>>>> Ben
>>>>
>>>
>>
>