[suspend/resume] xbd and xennet

[Jean-Yves Migeon <jean-yves.migeon%espci.fr@localhost>]

Hi,

Here's a small report relating to suspend/resume progress for port-xen. 
Work is going well, now that some hard-to-find issues have been solved. 
My current local repo can successfully suspend and resume back with a 
working console and xenstore, which makes debugging a lot easier.

I am on my way for xbd and xennet suspend/resume, but I am stuck on the 
right way to do it. xentools, xenconsole and xenstore being low level 
interfaces, they are closely tied to Xen's API, their implementation 
tend to be same from one OS to another (albeit some differences, but no 
big deal). For xbd and xennet, things are a little bit different though.

Before asking questions, I will first present how devices work in Xen 
world. Please correct if there are any mistakes, as I am currently 
basing my xbd/xennet stuff on these assumptions.

Devices are communication end points between two domains. It usually 
happens between a privileged (dom0 or a driver-domain) and unprivileged 
domain (the domU). On dom0's side, the end point is called the 
"backend". On domU's side, it is called the frontend. Both communicates 
by means of rings, similar to xenstore and xencons. There are two rings: 
one ring is for frontend => backend (writing a block or sending a 
packet), and another ring for backend => frontend (reading a block or 
receiving a packet).

At start of day (we consider that xencons and xenstore are set up), 
backend and frontend are initialized like below (information are 
exchanged between dom0 and domU through xenstore, with key-value pairs - 
see arch/xen/xenbus/xenbus_xs.c).

Note that during xenbus attach phase, in autoconf, domU is creating a 
thread (see xenbus_attach) which creates two watches, one for frontend 
modification (on device node), the other for backend modification (on 
backend node), in xenstore hierarchy.

1 - dom0 populates xenstore nodes, by placing some values required by 
frontend to initialize its driver. Entry added is "vbd", under the path 
associated to the domU's devices (the device node). The entry is 
following (assuming our domU has domid = 1):
portable-current# xenstore-ls /local/domain/1/device
vbd = ""
1 = ""
 backend-id = "0"
 backend = "/local/domain/0/backend/vbd/1/1"
...

(note to self: when a domain is making requests to xenstore with 
relative paths, xenstore considers that the domain requests data about 
himself - that way, a domain does not need to know its domid to get its 
own hierarchy information).

2 - frontend's watch fires. calls frontend_changed(), then 
xenbus_probe_frontends(). It parses the node's content (termed a 
directory, since xenstore is like a very simple fs), gets the device's 
type, puts the frontend's entry "state" in "XenbusStateInitialising", 
then process to the block device initialisation (in our case, 
xbd_xenbus_attach() ). backend is idle here.

3 - during attach, domU allocate a page, initialize ring and private 
data structures, then offer the page through grant_table, and obtain the 
associated grant ref. This is where the "backend-id" is needed, to offer 
the page to backend. An unbound event channel is created, and both 
information are stored in xenstore ("ring-ref" and "event-channel"). 
Once done, domU sets the vbd state to XenbusStateInitialised, and 
install a watch for the backend's state.

4 - dom0's backend (via its watch) now sees that the frontend's state 
reached XenbusStateInitialised. It gets the grant reference and event 
channel from xenstore, and proceed to the mapping of the granted share, 
and the connection to the domU's port. Once done, it sets its state to 
XenbusStateConnected (this time, under /local/domain/0/backend/vbd/1/1, 
which is the backend's node in xenstore).

5 - domU's sees the new backend's state, then reads the device's info 
(size and sector-size, type). Proceed its initialisation, and put itself 
in XenbusStateConnected mode (in /local/domain/1/device/vbd/1).

6 - Communication starts.

[...]

7 => end : same happens during detach code. Backend's state gets to 
XenbusStateClosing, which is noticed by frontend's watch; domU starts 
graceful shutdown operations, then once finished, sets its state to 
XenbusStateClosed.

I skipped the backend XenbusInitWait state, which happens during step 1. 
From a suspend/resume perspective, it does not change much.

What differs during suspend/resume is that MFN may change (which means 
that the grant table mapping has to be done again), and that we do not 
go from a Unknown to a Connected state, and finally to a Closed one. 
Devices stay Connected, but their operation is halted during suspend.

Now, questions: depending on implementation, suspend and resume is not 
handled the same way for different OS's.
- linux (the XenSource 2.6.18 implementation) does not have a suspend 
routine for its devices. Instead, it checks during resume if anything 
has change, and resume operations by copying the ones left behind while 
suspending (from a shadow store).
- opensolaris takes a more traditional route, with a suspend and a 
resume routine.

I imagine that they have pros and cons. Both approaches are different 
though, and before writing the suspend/resume routines, I would like to 
know which one should be taken for netbsd. The linux way may have 
minimal overhead, but requires extra precaution during resume. Solaris' 
way looks cleaner, as we could share code between traditional autoconf 
routines and suspend/resume. But it may add extra code paths we do not 
necessarily require (like freeing bufs during suspend and reallocate 
them during resume).

So I would like to have your opinion.

BTW, where does the XPMAP_OFFSET (difference between KERNBASE and 
KERNTEXTOFF) stem from? I thought that kernel virtual space always 
started from KERNBASE address? Any reason why Xen domUs require it to 
start from KERNTEXTOFF rather than KERNBASE?

Anyway, thanks for reading :)

Cheers,

--
Jean-Yves Migeon
jean-yves.migeon%espci.fr@localhost