pkgsrc-Bugs archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

pkg/57445: firefox crashes on startup



	Note: There was a bad value `toolchain' for the field `Class'.
	It was set to the default value of `sw-bug'.

>Number:         57445
>Category:       pkg
>Synopsis:       firefox crashes on startup
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    pkg-manager
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon May 29 12:30:00 +0000 2023
>Originator:     Martin Husemann
>Release:        NetBSD 10.99.4
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD martins.aprisoft.de 10.99.4 NetBSD 10.99.4 (GENERIC) #171: Sat May 27 10:59:49 CEST 2023 martin%martins.aprisoft.de@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
Architecture: x86_64
Machine: amd64
>Description:

This has been talked about on the mailing list on and off with no commitable
solution - so it is time to file a PR.

The bug described below *might* be related to the TLS issue reported in
PR 50277, and actually disabling TLS use in lib mesa seems to help, but
it is not clear what this means - could be just different timing helping
accidently/by chance.

So what happens when I start firefox: firefox is supposed to create a render
thread and use libmesa for that. It creates a GL context and sets it for
that thread. When something goes wrong, the error code is fetched from
that context.

However, at regular startup on my machine something goes wrong but the GL
context has not been set (is still NULL) and this NULL dereference (to get
the last error status) kills firefox.

Note: upstream (in libmesa) the context never is NULL, it is initialized to
the address of a static (empty) &dummyContext.

The change to make it initally NULL is a NetBSD specific hack to work around
some ld.elf_so restriction (that I don't fully understand).

The bug is obscured slightly by:
 - gdb/ptrace not working well together with -current ld.elf_so, TLS, massive
   threaded apps like firefox. Running firefox from gdb often makes it work
   but also when it doesn't work changes the details of the crash completely
   and also crashes gdb
 - firefox helpfully installing a signal handler catching the crash

So what I did is:
 - build firefox with options debug-info
   (see https://wiki.NetBSD.org/tutorials/pkgsrc/debugging_firefox/)
 - modify mesalib.old (see patch below)
 - run firefox w/o gdb and check the core
 - run firefox from inside gdb with a breakpoint on __glXSetCurrentContext
   which is the only place where the context ever gets changed

To me it looks like something goes wrong in the NetBSD variant of
Mesa's	GET_CURRENT_CONTEXT(ctx)
macro. Theoretically it is impossible to get ctx == NULL.

Within Firefox there should be a valid context in the render thread,
but it seems in the crash case it does not get that far (why it fails
to do so is still to be analyzed, but maybe anything including bad luck/timing).
It could (silently) happen on other systems too, but probably goes unnoticed
there as the crashing call just retrieves the error code from the dummyContext.


So here are the gdb sessions, one showing the actual crash from a core
dump, and one showing a successfull call to __glXSetCurrentContext
with a valid context.

[/usr/pkgobj/www/firefox/work/build/dist/bin] martin@martins > ./run-mozilla.sh ./firefox
Crash Annotation GraphicsCriticalError: |[0][GFX1-]: glxtest: cannot access /sys/bus/pci (t=0.385433) [GFX1-]: glxtest: cannot access /sys/bus/pci
ATTENTION: default value of option mesa_glthread overridden by environment.
libEGL warning: DRI2: failed to authenticate
ATTENTION: default value of option mesa_glthread overridden by environment.
_mesa_GetError() in thread 0x73016b7dd400 lwp 27206 with NULL context
Segmentation fault (core dumped)

 > gdb ./firefox firefox.core 
[..]
Reading symbols from ./firefox...
[New process 27206]
[..]
[New process 11054]
[New process 21185]
Core was generated by `firefox'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000730165aadb97 in _mesa_GetError ()
   from /usr/X11R7/lib/modules/dri/swrast_dri.so
[Current thread is 1 (process 27206)]

(gdb) bt
#0  0x0000730165aadb97 in _mesa_GetError ()
   from /usr/X11R7/lib/modules/dri/swrast_dri.so
#1  0x0000730179027166 in mozilla::gl::GLContext::InitImpl (
    this=0x73015f23f800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContext.cpp:526
#2  0x000073017902859a in mozilla::gl::GLContext::InitImpl (
    this=0x73015f23f800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContext.cpp:372
#3  mozilla::gl::GLContext::Init (this=0x73015f23f800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContext.cpp:324
#4  0x0000730179028612 in mozilla::gl::GLContextEGL::Init (this=0x73015f23f800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderEGL.cpp:412
#5  0x0000730179028a34 in mozilla::gl::GLContextEGL::CreateGLContext (egl=..., 
    desc=..., surfaceConfig=0x73015f29db40, surface=surface@entry=0x0, 
    useGles=useGles@entry=false, contextConfig=<optimized out>, 
    out_failureId=out_failureId@entry=0x73016b749700)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderEGL.cpp:759
#6  0x00007301790291d9 in mozilla::gl::GLContextEGLFactory::CreateImpl (
    aWindow=aWindow@entry=0x0, 
    aHardwareWebRender=aHardwareWebRender@entry=true, 
    aUseGles=aUseGles@entry=false)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderEGL.cpp:294
#7  0x0000730179029647 in mozilla::gl::GLContextEGLFactory::Create (
    aWindow=0x0, aHardwareWebRender=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderEGL.cpp:328
#8  0x000073017902968b in mozilla::gl::GLContextProviderEGL::CreateForCompositorWidget (aCompositorWidget=aCompositorWidget@entry=0x0, 
    aHardwareWebRender=aHardwareWebRender@entry=true)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderEGL.cpp:1003
#9  0x00007301792a9212 in CreateGLContextEGL ()
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderThread.cpp:1363
#10 0x00007301792a9487 in CreateGLContext (aError=...)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderThread.cpp:1396
#11 mozilla::wr::RenderThread::CreateSingletonGL (
    this=this@entry=0x73016ed67e00, aError=...)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderThread.cpp:1142
#12 0x00007301792a9a7d in mozilla::wr::RenderThread::InitDeviceTask (
    this=0x73016ed67e00)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderThread.cpp:973
#13 mozilla::wr::RenderThread::InitDeviceTask (this=0x73016ed67e00)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderThread.cpp:962
#14 0x0000730178c97dbe in mozilla::detail::runnable_args_base<(mozilla::detail::RunnableResult)0>::Run (this=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/dom/media/webrtc/transport/runnable_utils.h:41
#15 0x000073017880d552 in nsThread::ProcessNextEvent (this=0x73016edf9080, 
    aMayWait=<optimized out>, aResult=0x73016b749cd7)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/xpcom/threads/nsThread.cpp:1233
#16 0x0000730178801db1 in NS_ProcessNextEvent (aThread=<optimized out>, 
    aThread@entry=0x73016edf9080, aMayWait=aMayWait@entry=true)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/xpcom/threads/nsThreadUtils.cpp:477
#17 0x0000730178ce8d58 in mozilla::ipc::MessagePumpForNonMainThreads::Run (
    this=0x73016b4b50c0, aDelegate=0x73016b749d90)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/glue/MessagePump.cpp:330
#18 0x0000730178cad506 in MessageLoop::RunInternal (
    this=0x730182b46380 <__stack_chk_guard>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/chromium/src/base/message_loop.cc:381
#19 MessageLoop::RunHandler (this=0x730182b46380 <__stack_chk_guard>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/chromium/src/base/message_loop.cc:374
#20 MessageLoop::Run (this=this@entry=0x73016b749d90)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/chromium/src/base/message_loop.cc:356
#21 0x000073017880cd68 in nsThread::ThreadFunc (aArg=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/xpcom/threads/nsThread.cpp:391
#22 0x00007301748297a6 in _pt_root () from /usr/pkg/lib/nspr/libnspr4.so
#23 0x0000730182d7c2df in pthread__create_tramp (cookie=0x73016b7dd400)
    at /usr/src/lib/libpthread/pthread.c:592


[/usr/pkgobj/www/firefox/work/build/dist/bin] martin@martins > ./run-mozilla.sh -g ./firefox
Reading symbols from ./firefox...
(gdb) break __glXSetCurrentContext
Function "__glXSetCurrentContext" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (__glXSetCurrentContext) pending.
(gdb) run
[..]
[New LWP 15641 of process 12660]
Crash Annotation GraphicsCriticalError: |[0][GFX1-]: No GPUs detected via PCI (t=37.547) [GFX1-]: No GPUs detected via PCI
Crash Annotation GraphicsCriticalError: |[0][GFX1-]: No GPUs detected via PCI (t=37.547) |[1][GFX1-]: glxtest: process failed (received signal 5) (t=37.5471) [GFX1-]: glxtest: process failed (received signal 5)
[New LWP 4400 of process 12660]
[..]
[New LWP 20536 of process 12660]
[New LWP 22439 of process 12660]
[Switching to LWP 25340 of process 12660]

Thread 61 "Renderer" hit Breakpoint 1, __glXSetCurrentContext (
    c=0x7f1d1395c000)
    at /usr/xsrc/external/mit/MesaLib.old/dist/src/glx/glxcurrent.c:102
102	fprintf(stderr, "__glXSetCurrentContext(%p) in thread %p lwp %ld\n",

(gdb) p *c
$1 = {buf = 0x0, pc = 0x0, limit = 0x0, bufEnd = 0x0, bufSize = 0, 
  vtable = 0x7f1d26b24340, xid = 35651646, share_xid = 0, screen = 0, 
  psc = 0x7f1d2c8d3f00, imported = 0 '\000', currentContextTag = 4294967295, 
  renderMode = 0, feedbackBuf = 0x0, selectBuf = 0x0, fillImage = 0x0, 
  attributes = {stack = {0x0 <repeats 16 times>}, stackPointer = 0x0}, 
  error = 0, isDirect = 1, currentDpy = 0x7f1d39b31000, 
  currentDrawable = 35651644, vendor = 0x0, renderer = 0x0, version = 0x0, 
  extensions = 0x0, maxSmallRenderCommandSize = 0, majorOpcode = 151, 
  config = 0x7f1d268b5c00, currentReadable = 35651644, 
  client_state_private = 0x0, renderType = 32788, server_major = 0, 
  server_minor = 0, thread_refcount = 1, noError = 0, 
  gl_extension_bits = '\000' <repeats 16 times>}
(gdb) bt
#0  __glXSetCurrentContext (c=0x7f1d1395c000)
    at /usr/xsrc/external/mit/MesaLib.old/dist/src/glx/glxcurrent.c:102
#1  MakeContextCurrent (dpy=0x7f1d39b31000, draw=35651644, read=35651644, 
    gc_user=0x7f1d1395c000)
    at /usr/xsrc/external/mit/MesaLib.old/dist/src/glx/glxcurrent.c:253
#2  0x00007f1d32df0f75 in mozilla::gl::GLXLibrary::fMakeCurrent (
    context=<optimized out>, drawable=<optimized out>, 
    display=<optimized out>, this=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLXLibrary.h:68
#3  mozilla::gl::GLContextGLX::MakeCurrentImpl (this=0x7f1d138d4800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:473
#4  0x00007f1d32dff586 in mozilla::gl::GLContext::MakeCurrent (
    this=0x7f1d138d4800, aForce=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContext.cpp:2440
#5  0x00007f1d32e2858e in mozilla::gl::GLContext::InitImpl (
    this=0x7f1d138d4800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContext.cpp:372
#6  mozilla::gl::GLContext::Init (this=this@entry=0x7f1d138d4800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContext.cpp:324
#7  0x00007f1d32df328f in mozilla::gl::GLContextGLX::Init (this=0x7f1d138d4800)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:453
#8  operator() (__closure=__closure@entry=0x7f1d1c6c84b0, attribs=...)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:351
#9  0x00007f1d32df36f6 in mozilla::gl::GLContextGLX::CreateGLContext (
    desc=..., display=..., drawable=<optimized out>, drawable@entry=35651644, 
    cfg=<optimized out>, cfg@entry=0x7f1d268b5c00, 
    ownedPixmap=<optimized out>, ownedPixmap@entry=0)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:417
#10 0x00007f1d32df3a76 in mozilla::gl::CreateForWidget (
    aXDisplay=aXDisplay@entry=0x7f1d39b31000, 
    aXWindow=aXWindow@entry=35651644, 
    aHardwareWebRender=aHardwareWebRender@entry=true, 
    aForceAccelerated=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:620
#11 0x00007f1d32df3c62 in mozilla::gl::CreateForWidget (
    aForceAccelerated=<optimized out>, aHardwareWebRender=<optimized out>, 
    aXWindow=35651644, aXDisplay=0x7f1d39b31000)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:634
#12 mozilla::gl::GLContextProviderGLX::CreateForCompositorWidget (
    aCompositorWidget=<optimized out>, aHardwareWebRender=<optimized out>, 
    aForceAccelerated=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderGLX.cpp:634
#13 0x00007f1d32df42b7 in mozilla::gl::GLContextProviderLinux::CreateForCompositorWidget (aCompositorWidget=<optimized out>, 
    aHardwareWebRender=aHardwareWebRender@entry=true, 
    aForceAccelerated=aForceAccelerated@entry=true)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/gl/GLContextProviderLinux.cpp:29
#14 0x00007f1d330a9d10 in mozilla::wr::RenderCompositorOGL::Create (
    aWidget=..., aError=...)
    at /usr/pkgobj/www/firefox/work/build/dist/include/mozilla/RefPtr.h:280
#15 0x00007f1d330baa17 in mozilla::wr::RenderCompositor::Create (aWidget=..., 
    aError=...)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderCompositor.cpp:223
#16 0x00007f1d330bfd50 in mozilla::wr::NewRenderer::Run (this=0x7f1d13972100, 
    aRenderThread=..., aWindowId=...)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/WebRenderAPI.cpp:71
#17 0x00007f1d330a604c in mozilla::wr::RenderThread::RunEvent (
    this=0x7f1d2c8d2c00, aWindowId=..., aEvent=...)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/gfx/webrender_bindings/RenderThread.cpp:526
#18 0x00007f1d330a122e in mozilla::detail::RunnableMethodArguments<mozilla::wr::WrWindowId, mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> >&&>::applyImpl<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> >), StoreCopyPassByConstLRef<mozilla::wr::WrWindowId>, StoreCopyPassByRRef<mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> > >, 0ul, 1ul> (args=..., m=<optimized out>, o=<optimized out>)
    at /usr/pkgobj/www/firefox/work/build/dist/include/nsThreadUtils.h:902
#19 mozilla::detail::RunnableMethodArguments<mozilla::wr::WrWindowId, mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> >&&>::apply<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> >)> (m=<optimized out>, 
    o=<optimized out>, this=<optimized out>)
    at /usr/pkgobj/www/firefox/work/build/dist/include/nsThreadUtils.h:1169
#20 mozilla::detail::RunnableMethodImpl<mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> >), true, (mozilla::RunnableKind)0, mozilla::wr::WrWindowId, mozilla::UniquePtr<mozilla::wr::RendererEvent, mozilla::DefaultDelete<mozilla::wr::RendererEvent> >&&>::Run (
    this=<optimized out>)
    at /usr/pkgobj/www/firefox/work/build/dist/include/nsThreadUtils.h:1216
#21 0x00007f1d3260d552 in nsThread::ProcessNextEvent (this=0x7f1d29ff7a00, 
    aMayWait=<optimized out>, aResult=0x7f1d1c6c8cd7)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/xpcom/threads/nsThread.cpp:1233
#22 0x00007f1d32601db1 in NS_ProcessNextEvent (aThread=<optimized out>, 
    aThread@entry=0x7f1d29ff7a00, aMayWait=aMayWait@entry=true)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/xpcom/threads/nsThreadUtils.cpp:477
#23 0x00007f1d32ae8d58 in mozilla::ipc::MessagePumpForNonMainThreads::Run (
    this=0x7f1d2683d0c0, aDelegate=0x7f1d1c6c8d90)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/glue/MessagePump.cpp:330
#24 0x00007f1d32aad506 in MessageLoop::RunInternal (
    this=0x7f1d3c9c8380 <__stack_chk_guard>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/chromium/src/base/message_loop.cc:381
#25 MessageLoop::RunHandler (this=0x7f1d3c9c8380 <__stack_chk_guard>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/chromium/src/base/message_loop.cc:374
#26 MessageLoop::Run (this=this@entry=0x7f1d1c6c8d90)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/ipc/chromium/src/base/message_loop.cc:356
#27 0x00007f1d3260cd68 in nsThread::ThreadFunc (aArg=<optimized out>)
    at /usr/pkgobj/www/firefox/work/firefox-112.0.1/xpcom/threads/nsThread.cpp:391
#28 0x00007f1d2e6297a6 in _pt_root () from /usr/pkg/lib/nspr/libnspr4.so
#29 0x00007f1d3cbfe2df in pthread__create_tramp (cookie=0x7f1d268dc400)
    at /usr/src/lib/libpthread/pthread.c:592




>How-To-Repeat:
On my machine: just try to start firefox.

>Fix:
n/a

Patch used against mesalib.old:

Index: dist/src/glx/glxcurrent.c
===================================================================
RCS file: /cvsroot/xsrc/external/mit/MesaLib.old/dist/src/glx/glxcurrent.c,v
retrieving revision 1.1.1.2
diff -u -p -r1.1.1.2 glxcurrent.c
--- dist/src/glx/glxcurrent.c	11 Jul 2021 20:36:29 -0000	1.1.1.2
+++ dist/src/glx/glxcurrent.c	29 May 2023 11:29:27 -0000
@@ -34,6 +34,7 @@
  */
 
 #include <pthread.h>
+#include <lwp.h>
 
 #include "glxclient.h"
 #include "glapi.h"
@@ -98,6 +99,8 @@ __thread void *__glX_tls_Context __attri
 _X_HIDDEN void
 __glXSetCurrentContext(struct glx_context * c)
 {
+fprintf(stderr, "__glXSetCurrentContext(%p) in thread %p lwp %ld\n",
+  c, pthread_self(), (long)_lwp_self());
    __glX_tls_Context = (c != NULL) ? c : &dummyContext;
 }
 
Index: dist/src/mesa/main/getstring.c
===================================================================
RCS file: /cvsroot/xsrc/external/mit/MesaLib.old/dist/src/mesa/main/getstring.c,v
retrieving revision 1.1.1.2
diff -u -p -r1.1.1.2 getstring.c
--- dist/src/mesa/main/getstring.c	11 Jul 2021 20:36:32 -0000	1.1.1.2
+++ dist/src/mesa/main/getstring.c	29 May 2023 11:29:27 -0000
@@ -24,6 +24,7 @@
 
 
 #include <stdbool.h>
+#include <lwp.h>
 #include "glheader.h"
 #include "context.h"
 #include "debug_output.h"
@@ -325,6 +326,11 @@ GLenum GLAPIENTRY
 _mesa_GetError( void )
 {
    GET_CURRENT_CONTEXT(ctx);
+
+if (ctx == NULL)
+  fprintf(stderr, "_mesa_GetError() in thread %p lwp %ld with NULL context\n",
+    pthread_self(), (long)_lwp_self());
+
    GLenum e = ctx->ErrorValue;
    ASSERT_OUTSIDE_BEGIN_END_WITH_RETVAL(ctx, 0);
 



Home | Main Index | Thread Index | Old Index