Wine automated testing update

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Wine automated testing update

Jeremy White
Hi Folks,

At the last Wine conference, I volunteered to find a home for the
WineTestBot that Ge's brother has been hosting, and to take over the
Buildbot that Dan and Austin have been so diligently maintaining.

This is an update on that project, mostly to help keep my
procrastinating self moving it along.

First, we have a lovely set of hardware; 4 servers, in a dedicated rack,
with a dedicated static ip to the outside world.  One server is the
gateway, 2 are dedicated Linux test systems (with nice fast processor +
gpus), and one is a beefy system capable of running many VMs.

The plan was to move the existing VMWare based WineTestBot to this
hardware and simply get the existing system up and running on the new
hardware.

Here we hit our first snag; the VMWare folks are not willing to provide
a permanent license for VMWare to us.

So, we've shifted gears, and are exploring whether something like qemu +
kvm would be a sufficient alternate.  I've done a little of this work,
and Maarten is hopefully going to help me try converting the existing
VMs to see if we can save any hassle by doing a conversion.

In parallel, I learned a great deal about buildbot and the buildbot
system that Dan has built.

There are two especially valuable and useful functions provided by this
buildbot:  the first is essentially a patch watcher; a process that
catches simple mistakes (e.g. build errors on different architectures,
and so on).  The second is more powerful and is encapsulated by the
'dotests.sh' shell script.

The dotests.sh script does a good job of basically running make test and
noticing any test failures outside of 'the usual ones'.  There is an
infrastructure for

Buildbot is a good tool, and I invested a lot of energy into it (as have
Dan and Austin).

However, Alexandre has persuaded me that we should first explore
integrating the dotest.sh functionality into the existing WineTestBot,
as that would allow us to have a very simple clean web page, and would
also allow us to integrate most cleanly with Alexandre's back end tools.

Of course, WineTestBot is Perl code, and my Perl fu is low.  So I have
recently asked Francois to help with that effort as well.

So the basic plan is as follows:

  1.  Test qemu/kvm with a few Windows versions to see if it'll work.

  2.  If so, modify WineTestBot to use qemu instead of vmware.
        (This is 'easy' because the WineTestBot code nicely isolates
        the VM functions, but 'hard', because VMWare provides some
        APIs that may have to be replicated).

  3.  Migrate WineTestBot to the new hardware stack

  4.  Write new code to run 'dotests.sh' as part of WineTestBot

Cheers,

Jeremy


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Lei Zhang-2
On Wed, Feb 1, 2012 at 12:11 PM, Jeremy White <[hidden email]> wrote:
>  1.  Test qemu/kvm with a few Windows versions to see if it'll work.

I've had good luck running 32-bit Windows in qemu + kvm. You'll
probably need to add -rtc=localtime. If you plan to run it headless
and connect via the build in vnc, add -usbdevice tablet.


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

John Klehm
For what it's worth VirtualBox has always treated me well virtualizing
XP on Win7 and Ubuntu 11.10  as well.

--John

On Wed, Feb 1, 2012 at 11:27 PM, Lei Zhang <[hidden email]> wrote:
> On Wed, Feb 1, 2012 at 12:11 PM, Jeremy White <[hidden email]> wrote:
>>  1.  Test qemu/kvm with a few Windows versions to see if it'll work.
>
> I've had good luck running 32-bit Windows in qemu + kvm. You'll
> probably need to add -rtc=localtime. If you plan to run it headless
> and connect via the build in vnc, add -usbdevice tablet.
>
>


Reply | Threaded
Open this post in threaded view
|

re: Wine automated testing update

Dan Kegel
In reply to this post by Jeremy White
Jeremy wrote:
> the VMWare folks are not willing to provide a permanent license for VMWare to us.
>
> So, we've shifted gears, and are exploring whether something like qemu +
> kvm would be a sufficient alternate.

What's the plan for automated MacOSX testing?   I hear 10.7
supports running in a VM...


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Jeremy White-2
On 02/03/2012 10:47 AM, Dan Kegel wrote:
> Jeremy wrote:
>> the VMWare folks are not willing to provide a permanent license for VMWare to us.
>>
>> So, we've shifted gears, and are exploring whether something like qemu +
>> kvm would be a sufficient alternate.
>
> What's the plan for automated MacOSX testing?   I hear 10.7
> supports running in a VM...

No plan at all, which is an oversight.

I think we may have to have a 'real' Windows box to do any kind of
credible D3D tests; perhaps we have to make the same argument for Mac
OSX.  But boy it sure would be nice if we could put MacOSX into a VM...

Cheers,

Jeremy


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Erich E. Hoover
In reply to this post by Jeremy White
On Wed, Feb 1, 2012 at 1:11 PM, Jeremy White <[hidden email]> wrote:
> Hi Folks,
>
> At the last Wine conference, I volunteered to find a home for the
> WineTestBot that Ge's brother has been hosting, and to take over the
> Buildbot that Dan and Austin have been so diligently maintaining.
> ...

Is there any chance that as part of this move we can get a virtual
network adapter setup on the test bot?  I'd like to add some tests
that require more than one network card to do properly.

Erich Hoover
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Charles Davis-3
In reply to this post by Jeremy White-2

On Feb 3, 2012, at 10:13 AM, Jeremy White wrote:

> On 02/03/2012 10:47 AM, Dan Kegel wrote:
>> Jeremy wrote:
>>> the VMWare folks are not willing to provide a permanent license for VMWare to us.
>>>
>>> So, we've shifted gears, and are exploring whether something like qemu +
>>> kvm would be a sufficient alternate.
>>
>> What's the plan for automated MacOSX testing?   I hear 10.7
>> supports running in a VM...
>
> No plan at all, which is an oversight.
>
> I think we may have to have a 'real' Windows box to do any kind of
> credible D3D tests; perhaps we have to make the same argument for Mac
> OSX.  But boy it sure would be nice if we could put MacOSX into a VM...
Yeah... Trouble is, you need to run it on a Mac anyway. (The host OS doesn't have to be Mac OS, but it has to be running on Mac hardware.) At least, that's my understanding of Apple's SLA for Mac OS X.

I have a launchd.plist (a config file for the Mac OS launcher daemon, their version of init, inetd, and cron all rolled into one) and a set of scripts that runs the tests every night after AJ commits. I've attached them here. The scripts expect to be run from the directory containing the Wine sources. The configuration is specific to my system (i.e. all the paths refer to directories in my home folder)--if anyone wants to use it, you'll have to change it for your system. Also note that the log file it produces needs to be turned over manually--this is because launchd handles setting the stdout and stderr paths to this file, and by the time my scripts can get around to removing it, the file is already open. Finally, when you activate the script, you must remember to set the DISPLAY variable in launchd's environment (launchctl setenv DISPLAY "$DISPLAY").

By the way, I still want to make my box into a buildslave. Though now I'm thinking about making my old MacBook Pro into a slave, instead of my Mac Pro which gets used a lot nowadays.

Chip


>
> Cheers,
>
> Jeremy
>
>




wine-test-mac.tar.bz2 (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Francois Gouget
In reply to this post by Jeremy White
On Wed, 1 Feb 2012, Jeremy White wrote:
[...]
> However, Alexandre has persuaded me that we should first explore
> integrating the dotest.sh functionality into the existing WineTestBot,
> as that would allow us to have a very simple clean web page, and would
> also allow us to integrate most cleanly with Alexandre's back end tools.
>
> Of course, WineTestBot is Perl code, and my Perl fu is low.  So I have
> recently asked Francois to help with that effort as well.
>
> So the basic plan is as follows:
[...]
>   2.  If so, modify WineTestBot to use qemu instead of vmware.
>         (This is 'easy' because the WineTestBot code nicely isolates
>         the VM functions, but 'hard', because VMWare provides some
>         APIs that may have to be replicated).

So I have finally found the time to get started on this. Yay!
It's not done yet but I figure it's time for me to give a status report
and ask some questions.


--- LibVirt

I updated the Perl scripts to remove the dependency on VMware. Now I use
libvirt which is a virtualization API that supports multiple
virtualization technologies such as KVM/QEMU, but also VirtualBox, Xen
and even VMware. libvirt also makes it possible to control VMs on remote
boxes so we can distribute the load over multiple machines. For more
details see the project's web site:

   http://libvirt.org/


--- Host / VM communication

One piece that's missing is the ability to copy files from/to a VM and
to run specific commands in a VM. This was provided by VMware's VIX API
combined with a guest agent. See VMCopyFileFromHostToGuest() and
VM::RunScriptInGuest() on the following page for instance:
http://www.vmware.com/support/developer/vix-api/vix111_reference/index2.html

All replacements involve installing some application in the Windows VM.
So given that we want to run Winetest too I have the following
requirements:
 * The application should run on everything from Windows 8 down to NT4.
 * It should not bring in dependencies that would modify the behavior of
   Windows. So anything that depends on installing .Net or upgrading
   standard dlls located in c:\windows\system32 is out.
 * Ideally it should also have a 64bit version so we can have
   64bit-only Windows VMs.
So I looked for a replacement but came away not entirely satisfied:

 * There is some talk of a QEmu Guest Agent but as far as I can tell
   it's just vaporware. If someone knows more I'd be interested (even if
   it would tie us to QEmu).
   http://wiki.qemu.org/Features/QAPI/GuestAgent

 * There is the BuildBot slave which probably provides the functionality
   we need. But it is implemented in Python which only works on Windows
   XP and later, and comes as an MSI installer. I did not check exactly
   how to interface with it.
   http://trac.buildbot.net/wiki/RunningBuildbotOnWindows

 * There's Cygwin's OpenSSH server which would give us what we need
   through scp and ssh. Cygwin does not install on NT4 and I'm not
   convinced it's really free of side-effects. In particular copying
   the cygwin directory to another computer does not seem to work.
   http://cygwin.com/install.html

 * There's more limited OpenSSH installations derived from Cygwin
   but I'm not sure they're much better on the side-effects side.
   http://sshwindows.sourceforge.net/download/

 * For my own Windows VMs (see the fg-xxx VMs on test.winehq.org), what
   I use is a Windows implementation of netcat.
   http://joncraton.org/blog/46/netcat-for-windows

   I essentially run it in a batch that does:
      REM Receive winetest and how to invoke it
      nc95 -l -p%port% >app.exe
      nc95 -l -p%port% >cmdline.bat
      REM Run winetest, sending its output to the client
      nc95 -l -p%port% -e .\cmdline.bat

   I'm not entirely satisfied with it (and not because netcat is
   totally insecure, the VMs are behind firewalls anyway). Rather it's
   because it's not very flexible and because it has race conditions
   (which I work around with pauses on the client side). I could make it
   more flexible with something like:
     :start
     nc95 -l -p%port1% >cmd.bat
     nc95 -l -p%port1% -e .\cmd.bat
     goto start
   And put more nc95 commands running on a second port in cmd.bat.

 * Modified Windows netcat. I'd like to be able to run a netcat server
   so that on the client side I could do:
   (
      echo "write c:/winetest.exe"
      cat winetest.exe
   ) | netcat $host $port
   echo "run c:/winetest.exe my options here" |
       netcat $host $port >winetest.log

   The server would see the command on the first line and do the
   relevant action.


--- Network trouble

I'm having trouble getting the network to work fine in the QEmu VMs.
   
Eventually I discovered that if I bridged to eth0 then it's just the
communication between the host and the VM that does not work.
Communication with other machines on the LAN or with the Internet works.
However that's going to be a big problem to send the jobs to the VMs
(see previous section, plus the dhcp and dns is also running on the host
in my case).
     
Now, rather than bridge to eth0 what I'd want on my machine is to bridge
to tap0, a local network where I have all my VMware VMs.
   
What I typically see if I ping the host from the VM is that the echo
requests go out and the echo replies are sent. But ping on the VM sees
nothing. It could be a firewalling issue though my VMware VMs on the
same network work fine.


--- Patches

My code is far from ready for commits but I stumbled upon a few fixes.
Where should I send those?


--- Build VM

The installation instructions mention setting up a 'Wine build VM' but
are short on details. Is it actually supposed to be a VM. What name?
Identified through the 'Type' field in the VMs table?


--- Miscellaneous

 * In Engine.pl, HandleBuildNotification() calls a
   ${ProjectName}RetrieveBuild.pl script which concerns me because there
   is no such script. There is a WineRunBuild.pl script that could match
   though. Still that indicates the Git code may not be exactly what's
   running on the current WineTestBot. Does anyone know more?

 * The virtio disk and network drivers appear to be a must. Replacing
   the default vga emulation with the cirrus one makes it possible to  
   suspend / hibernate the VM (which is probably useless but still  
   instills more confidence to me). Switching to vmware seems to give  
   the best performance but Libvirt does not recognize it :-(
   It looks like the XML schema is simply being out of date. Does
   anyone know more?
   Still on the subject of optimizing Windows to work with QEmu, does
   anyone know if it's possible to expose the full capabilities of the
   graphics card to the VM? By declaring the graphics card as
   'passthrough' maybe? That would be nice for Direct3D but I'm not
   getting my hopes up :-(


--
Francois Gouget <[hidden email]>



Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Julius Schwartzenberg-4
Francois Gouget wrote:

> All replacements involve installing some application in the Windows VM.
> So given that we want to run Winetest too I have the following
> requirements:
>  * The application should run on everything from Windows 8 down to NT4.
>  * It should not bring in dependencies that would modify the behavior of
>    Windows. So anything that depends on installing .Net or upgrading
>    standard dlls located in c:\windows\system32 is out.
>  * Ideally it should also have a 64bit version so we can have
>    64bit-only Windows VMs.
> So I looked for a replacement but came away not entirely satisfied:

Maybe freeSSHd and/or freeFTPd are useful:
http://www.freesshd.com/

At least the webpage claims it runs on Windows NT4 and higher. Logging
in over SSH just gives a cmd.com command shell as far as I remember, so
this should allow for some interaction with the system running on the VM.

Best regards,
Julius


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Michael Stefaniuc-2
In reply to this post by Francois Gouget
Francois,

On 06/21/2012 04:21 PM, Francois Gouget wrote:

> --- Network trouble
>
> I'm having trouble getting the network to work fine in the QEmu VMs.
>    
> Eventually I discovered that if I bridged to eth0 then it's just the
> communication between the host and the VM that does not work.
> Communication with other machines on the LAN or with the Internet works.
> However that's going to be a big problem to send the jobs to the VMs
> (see previous section, plus the dhcp and dns is also running on the host
> in my case).
>      
> Now, rather than bridge to eth0 what I'd want on my machine is to bridge
> to tap0, a local network where I have all my VMware VMs.
can you please send me the network details of the host?
Normally you don't bridge to eth0 but create a bridge interface and eth0
is a member of that bridge aka eth0 won't have an IP associated with it
as the host IP would be on virtbr0.


> What I typically see if I ping the host from the VM is that the echo
> requests go out and the echo replies are sent. But ping on the VM sees
> nothing. It could be a firewalling issue though my VMware VMs on the
> same network work fine.

bye
        michael


Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

André Hentschel
In reply to this post by Francois Gouget
Am 21.06.2012 16:21, schrieb Francois Gouget:

> On Wed, 1 Feb 2012, Jeremy White wrote:
> [...]
>> However, Alexandre has persuaded me that we should first explore
>> integrating the dotest.sh functionality into the existing WineTestBot,
>> as that would allow us to have a very simple clean web page, and would
>> also allow us to integrate most cleanly with Alexandre's back end tools.
>>
>> Of course, WineTestBot is Perl code, and my Perl fu is low.  So I have
>> recently asked Francois to help with that effort as well.
>>
>> So the basic plan is as follows:
> [...]
>>   2.  If so, modify WineTestBot to use qemu instead of vmware.
>>         (This is 'easy' because the WineTestBot code nicely isolates
>>         the VM functions, but 'hard', because VMWare provides some
>>         APIs that may have to be replicated).
>
> So I have finally found the time to get started on this. Yay!
> It's not done yet but I figure it's time for me to give a status report
> and ask some questions.
> ...
> --- Patches
>
> My code is far from ready for commits but I stumbled upon a few fixes.
> Where should I send those?
>

I'd suggest to move the code at some point over to source.winehq.org with Francois as committer, so everybody can work on it in the normal ways.
Testbot became a must-have very quickly and needs improvements now, so having at in a central place just makes sense.

--

Best Regards, André Hentschel




Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Francois Gouget
In reply to this post by Francois Gouget

I have pushed my libvirt development branch to GitHub so you can see
what's brewing:

   https://github.com/fgouget/winetestbot/commits/libvirt
or
   git://github.com/fgouget/winetestbot.git


Note that I still rebase the libvirt branch regularly as I realize my
mistakes or want to reorganize things so they make sense. So know that
basing your work on my libvirt branch will be like trying to build a
house on quicksand.

This should settle down once WineTestBot is fully working on my test
environment. For now it's not quite the case yet: it's missing the code
to copy files to/from the VM and run stuff in the VM. But at least the
web site works, VMs get started/reverted and I can build tests in the
build VM (except for msvcp90 because of bug 30980 :-( ).

It turns out that out of the ~30 patches so far only one is for libvirt.
The others add documentation (the result of my trying to understand how
things work), remove dead code, tweak things a bit and include the odd
fix.

I have attached one patch that could be worth applying to the current
setup:
   web: Fix a couple of links to the registration page.


Maarten, can you confirm that what's in Git corresponds to the code that
runs the current production machines?


--
Francois Gouget <[hidden email]>



wtb-registration.diff (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Francois Gouget
In reply to this post by Julius Schwartzenberg-4
On Thu, 21 Jun 2012, Julius Schwartzenberg wrote:
[...]
> Maybe freeSSHd and/or freeFTPd are useful:
> http://www.freesshd.com/
>
> At least the webpage claims it runs on Windows NT4 and higher. Logging
> in over SSH just gives a cmd.com command shell as far as I remember, so
> this should allow for some interaction with the system running on the VM.

I came across it but had eliminated it because it does not seem to be
open-source. That might have been too harsh since it's free (as in
beer). So I gave it a try.

The problem is that when I start an application such as notepad, it does
not appear on the VM's desktop. I suspect freeSSHd creates a separate
window station and that's going to break Wine's tests. I looked for an
option to change this behavior but found none.


(the KpyM SSH Server has the same issue)

--
Francois Gouget <[hidden email]>



Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Saulius Krasuckas-2
In reply to this post by Francois Gouget
* On Thu, 21 Jun 2012, Francois Gouget wrote:
>
> --- Host / VM communication
>
> One piece that's missing is the ability to copy files from/to a VM and
> to run specific commands in a VM.
  ...
>  * There's Cygwin's OpenSSH server which would give us what we need
>    through scp and ssh. Cygwin does not install on NT4 and I'm not
>    convinced it's really free of side-effects. In particular copying
>    the cygwin directory to another computer does not seem to work.
>    http://cygwin.com/install.html

Seems like some guy Darren Tucker [1] is in the same boat:

" I have an old windows VM with an oldish cygwin that I use for the
regression tests.  Investigating one of the test failures, I see that
it's for UsePrivilegeSeparation=sandbox, and it seems to be because
setrlimit(RLIMIT_FSIZE, ...) is not supported.

IMO, this isn't a big loss, since the most useful thing in the rlimit
"sandbox" is the descriptor limits.  Can anyone see a reason not to just
omit RLIMIT_FSIZE on cygwin?  It's a no-op unless you've explicitly set
"UsePrivilegeSeparation sandbox" in sshd_config. "

Maybe you could ask him about his experience on side-effects?

OTOH, to measure the difference between modern Win and NT4 one could
probably compile whole env from the scratch and run Cygwin regression
tests.  Some guys are doing "make -s test". [2]

>  * There's more limited OpenSSH installations derived from Cygwin
>    but I'm not sure they're much better on the side-effects side.
>    http://sshwindows.sourceforge.net/download/

I've found tutorial about installing rsync on old Windows. [3]

Maybe rsync could look like having fewer side-effects (or no at all) with
this port now that it has been resurrected by some degree -- support for
Win7 x64 was added this year. [4]

S.

[1] http://marc.info/?l=openssh-unix-dev&m=134127510031310&w=2
[2] http://sourceforge.net/tracker/index.php?func=detail&aid=2993522&group_id=12694&atid=112694
[3] http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html
[4] http://sshwindows.webheat.co.uk/display/sshwindows/OpenSSH+for+Windows



Reply | Threaded
Open this post in threaded view
|

Re: Wine automated testing update

Julius Schwartzenberg-4
In reply to this post by Francois Gouget
Francois Gouget wrote:

> On Thu, 21 Jun 2012, Julius Schwartzenberg wrote:
> [...]
>> Maybe freeSSHd and/or freeFTPd are useful:
>> http://www.freesshd.com/
>>
>> At least the webpage claims it runs on Windows NT4 and higher. Logging
>> in over SSH just gives a cmd.com command shell as far as I remember, so
>> this should allow for some interaction with the system running on the VM.
>
> I came across it but had eliminated it because it does not seem to be
> open-source. That might have been too harsh since it's free (as in
> beer). So I gave it a try.
>
> The problem is that when I start an application such as notepad, it does
> not appear on the VM's desktop. I suspect freeSSHd creates a separate
> window station and that's going to break Wine's tests. I looked for an
> option to change this behavior but found none.

I just looked at this problem again and I found a solution. In case
you're still interested, there is a program called PsExec that allows
you to specify the session in which a program can be run.

It can be found here:
http://technet.microsoft.com/en-us/sysinternals/bb897553

When I execute this command when logged on with SSH using FreeSSHd,
notepad shows up on the VM's desktop:
PsExec.exe -i 0 notepad

It may be possible to use PsExec itself to do things remotely, but with
FreeSSHd it solves this issue.

Best regards,
Julius