Ok, I tried some tests today.<div>The hardware/software environment includes 2 cluster nodes (ESXi 4.1), 2Gb of RAM, 2 AMD Opteron 246 Processors (2GHz), trial version licenses. The opennebula installation is <u>self-contained</u>.</div>
<div>800MB of memory are used by the hypervisor itself (that info comes from vSphere Client) so only 1,2 GB are free, but OpenNebula seems unaware of that :-(</div>
<div><br></div><div><div><font face="'courier new', monospace">oneadmin@custom2:/srv/cloud/templates/vm$ onehost list</font></div><div><font face="'courier new', monospace"> ID NAME CLUSTER RVM TCPU FCPU ACPU TMEM FMEM STAT</font></div>
<div><font face="'courier new', monospace"> 2 <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> default 0 200 200 200 2G 0K on</font></div><div><font face="'courier new', monospace"> 1 <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> default 0 200 200 200 2G 0K on</font></div>
</div><div><br></div><div><div><font face="'courier new', monospace">oneadmin@custom2:/srv/cloud/templates/vm$ onehost show 1</font></div><div><font face="'courier new', monospace">HOST 1 INFORMATION </font></div>
<div><font face="'courier new', monospace">ID : 1 </font></div><div><font face="'courier new', monospace">NAME : <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> </font></div>
<div><font face="'courier new', monospace">CLUSTER : default </font></div><div><font face="'courier new', monospace">STATE : MONITORED </font></div>
<div><font face="'courier new', monospace">IM_MAD : im_vmware </font></div><div><font face="'courier new', monospace">VM_MAD : vmm_vmware </font></div>
<div><font face="'courier new', monospace">TM_MAD : tm_vmware </font></div><div><font face="'courier new', monospace"><br></font></div>
<div><font face="'courier new', monospace">HOST SHARES </font></div><div><font face="'courier new', monospace">MAX MEM : 2096460 </font></div>
<div><font face="'courier new', monospace">USED MEM (REAL) : 0 </font></div><div><font face="'courier new', monospace">USED MEM (ALLOCATED) : 0 </font></div>
<div><font face="'courier new', monospace">MAX CPU : 200 </font></div><div><font face="'courier new', monospace">USED CPU (REAL) : 0 </font></div>
<div><font face="'courier new', monospace">USED CPU (ALLOCATED) : 0 </font></div></div><div> </div><div>In each test I tried to start 3 VM using a nonpersistent image. The requirements of all of the three VM cannot be satisfied by a single cluster node.</div>
<div><br><div><font size="4">FIRST TEST:</font></div><div><br></div><div>The VM template for the first test is:</div><div><div><font face="'courier new', monospace">NAME<span style="white-space:pre-wrap"> </span>= "Debian Server"</font></div>
<div><font face="'courier new', monospace">CPU<span style="white-space:pre-wrap"> </span>= 1 </font></div><div><font face="'courier new', monospace">MEMORY<span style="white-space:pre-wrap"> </span>= 1024</font></div>
<div><font face="'courier new', monospace">OS<span style="white-space:pre-wrap"> </span>= [ ARCH = "i686" ]</font></div><div><font face="'courier new', monospace">DISK <span style="white-space:pre-wrap"> </span>= [IMAGE="Debian Server"]</font></div>
</div><div><font face="'courier new', monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">Only CPU and Memory info.</font></div><div><font face="arial, helvetica, sans-serif"><br>
</font></div><div>Here is the result:</div><div><div><font face="'courier new', monospace">oneadmin@custom2:/srv/cloud/templates/vm$ onevm list</font></div><div><font face="'courier new', monospace"> ID USER NAME STAT CPU MEM HOSTNAME TIME</font></div>
<div><font face="'courier new', monospace"> 66 oneadmin Debian S pend 0 0K 00 00:07:47</font></div><div><font face="'courier new', monospace"> 67 oneadmin Debian S pend 0 0K 00 00:07:45</font></div>
<div><font face="'courier new', monospace"> 68 oneadmin Debian S pend 0 0K 00 00:07:18</font></div></div><div><font face="'courier new', monospace"><br>
</font></div><div><font face="arial, helvetica, sans-serif">Forever in "pending" state... The VMs don't get scheduled</font></div><div><font face="arial, helvetica, sans-serif"><br>
</font></div><div><font face="arial, helvetica, sans-serif">oned.log doesn't report anything but resource polling informational messages.</font></div><div><font face="arial, helvetica, sans-serif">sched.log repeats this sequence:</font></div>
<div><div><font face="'courier new', monospace">Tue Feb 8 10:02:06 2011 [HOST][D]: Discovered Hosts (enabled): 1 2</font></div><div><font face="'courier new', monospace">Tue Feb 8 10:02:06 2011 [VM][D]: Pending virtual machines : 66 67 68</font></div>
<div><font face="'courier new', monospace">Tue Feb 8 10:02:06 2011 [RANK][W]: No rank defined for VM</font></div><div><font face="'courier new', monospace">Tue Feb 8 10:02:06 2011 [RANK][W]: No rank defined for VM</font></div>
<div><font face="'courier new', monospace">Tue Feb 8 10:02:06 2011 [RANK][W]: No rank defined for VM</font></div><div><font face="'courier new', monospace">Tue Feb 8 10:02:06 2011 [SCHED][I]: Select hosts</font></div>
<div><font face="'courier new', monospace"> PRI HID</font></div><div><font face="'courier new', monospace"> -------------------</font></div>
<div><font face="'courier new', monospace">Virtual Machine: 66</font></div><div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">Virtual Machine: 67</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">Virtual Machine: 68</font></div><div style="font-family:arial, helvetica, sans-serif">
<br></div></div><div><font face="arial, helvetica, sans-serif" size="4">SECOND TEST:</font></div><div><font face="arial, helvetica, sans-serif">VM template:</font></div><div>
<div style="font-family:arial, helvetica, sans-serif">NAME<span style="white-space:pre-wrap"> </span>= "Debian Server"</div><div style="font-family:arial, helvetica, sans-serif">VCPU<span style="white-space:pre-wrap"> </span>= 1 </div>
<div style="font-family:arial, helvetica, sans-serif">MEMORY<span style="white-space:pre-wrap"> </span>= 1024</div><div style="font-family:arial, helvetica, sans-serif">OS<span style="white-space:pre-wrap"> </span>= [ ARCH = "i686" ]</div>
<div style="font-family:arial, helvetica, sans-serif">DISK <span style="white-space:pre-wrap"> </span>= [IMAGE="Debian Server"]</div><div style="font-family:arial, helvetica, sans-serif">
<br></div><div style="font-family:arial, helvetica, sans-serif">Only VCPU and MEMORY info.</div><div style="font-family:arial, helvetica, sans-serif"><br></div><div style="font-family:arial, helvetica, sans-serif">
Results:</div><div><div><font face="'courier new', monospace"><div>oneadmin@custom2:/srv/cloud/templates/vm$ onevm list</div><div> ID USER NAME STAT CPU MEM HOSTNAME TIME</div>
<div> 76 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> 00 00:07:40</div><div> 77 oneadmin Debian S runn 0 0K <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> 00 00:07:38</div>
<div> 78 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> 00 00:05:58</div></font></div></div><div style="font-family:arial, helvetica, sans-serif"><br></div></div>
<div>Everything seems fine, but it's not since, as I said previously, each host has only 1.2 GB of memory free, so there's should be no space for two VMs on the same host.</div>
<div><br></div><div><div><font face="'courier new', monospace">oneadmin@custom2:/srv/cloud/templates/vm$ onehost list</font></div><div><font face="'courier new', monospace"> ID NAME CLUSTER RVM TCPU FCPU ACPU TMEM FMEM STAT</font></div>
<div><font face="'courier new', monospace"> 2 <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> default 2 200 200 200 2G 0K on</font></div><div><font face="'courier new', monospace"> 1 <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> default 1 200 200 200 2G 0K on</font></div>
<div><br></div><div>Both the hosts and the VMs report no useful info on the resource usage.</div><div>Logging to the VM of each console and executing "free -m" command I checked that every VM has 1GB of total memory allocated. So i decided to test the GB of memory on both VM at the same time using the utility called "memtester" which allocate a given amount of free memory using malloc and test it. The results reported memory access problems.</div>
<div><br></div><div>I decided here to go on and check if OpenNebula and VMWare ESXi fail to allocate VMs exceeding the resource capacity of the hosts, by starting two more VMs (requiring 1VCPU and 1GB memory each).</div>
<div>
Results:</div><div><div><font face="'courier new', monospace">oneadmin@custom2:~/var/79$ onevm list</font></div><div><font face="'courier new', monospace"> ID USER NAME STAT CPU MEM HOSTNAME TIME</font></div>
<div><font face="'courier new', monospace"> 76 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> 00 00:54:47</font></div><div><font face="'courier new', monospace"> 77 oneadmin Debian S runn 0 0K <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> 00 00:54:45</font></div>
<div><font face="'courier new', monospace"> 78 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> 00 00:53:05</font></div><div><font face="'courier new', monospace"> 79 oneadmin Debian S boot 0 0K <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> 00 00:10:22</font></div>
<div><font face="'courier new', monospace"> 80 oneadmin Debian S boot 0 0K <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> 00 00:09:47</font></div></div><div><br></div><div><div>
The new VM are allocated on custom7 machine (why???) but remain frozen on "boot" state.</div><div>That is a problem because those two new VM should not be allocated to any cluster node.</div></div><div><br></div>
<div><font size="4">THIRD TEST:</font></div><div>Here I followed Ruben suggestion...</div><div>The VM template:</div><div><div><font face="'courier new', monospace">oneadmin@custom2:/srv/cloud/templates/vm$ cat debian.vm </font></div>
<div><font face="'courier new', monospace">NAME<span style="white-space:pre-wrap"> </span>= "Debian Server"</font></div><div><font face="'courier new', monospace">CPU<span style="white-space:pre-wrap"> </span>= 1</font></div>
<div><font face="'courier new', monospace">VCPU<span style="white-space:pre-wrap"> </span>= 1 </font></div><div><font face="'courier new', monospace">MEMORY<span style="white-space:pre-wrap"> </span>= 1024</font></div>
<div><font face="'courier new', monospace">OS<span style="white-space:pre-wrap"> </span>= [ ARCH = "i686" ]</font></div><div><font face="'courier new', monospace">DISK <span style="white-space:pre-wrap"> </span>= [IMAGE="Debian Server"]</font></div>
</div><div><br></div><div>Both CPU/VCPU and MEMORY info.</div><div><br></div><div>Output with 3 VM:</div><div><div><font face="'courier new', monospace">oneadmin@custom2:~/var$ onevm list</font></div>
<div><font face="'courier new', monospace"> ID USER NAME STAT CPU MEM HOSTNAME TIME</font></div><div><font face="'courier new', monospace"> 81 oneadmin Debian S pend 0 0K 00 00:02:32</font></div>
<div><font face="'courier new', monospace"> 82 oneadmin Debian S pend 0 0K 00 00:02:30</font></div><div><font face="'courier new', monospace"> 83 oneadmin Debian S pend 0 0K 00 00:02:29</font></div>
</div><div><br></div><div>As in FIRST TEST the VMs don't get scheduled and remain in "pending" state.</div><div>sched.log repeats this message:</div><div><div><font face="'courier new', monospace">Tue Feb 8 12:00:05 2011 [HOST][D]: Discovered Hosts (enabled): 1 2</font></div>
<div><font face="'courier new', monospace">Tue Feb 8 12:00:05 2011 [VM][D]: Pending virtual machines : 81 82 83</font></div><div><font face="'courier new', monospace">Tue Feb 8 12:00:05 2011 [RANK][W]: No rank defined for VM</font></div>
<div><font face="'courier new', monospace">Tue Feb 8 12:00:05 2011 [RANK][W]: No rank defined for VM</font></div><div><font face="'courier new', monospace">Tue Feb 8 12:00:05 2011 [RANK][W]: No rank defined for VM</font></div>
<div><font face="'courier new', monospace">Tue Feb 8 12:00:05 2011 [SCHED][I]: Select hosts</font></div><div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>PRI<span style="white-space:pre-wrap"> </span>HID</font></div>
<div><font face="'courier new', monospace"><span style="white-space:pre-wrap"> </span>-------------------</font></div><div><font face="'courier new', monospace">Virtual Machine: 81</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">Virtual Machine: 82</font></div><div><font face="'courier new', monospace"><br>
</font></div><div><font face="'courier new', monospace">Virtual Machine: 83</font></div></div><div><br></div><div>Here I assumed that probably I should not declare the number of physical CPU in the VM template.</div>
<div><br></div><div>Another last test...</div><div><font size="4">FOURTH TEST:</font></div><div>Here I disabled an host, custom6, and started 3 VMs.</div><div>The VM template is the one that worked before:</div>
<div><div><font face="'courier new', monospace">oneadmin@custom2:/srv/cloud/templates/vm$ cat debian.vm </font></div><div><font face="'courier new', monospace">NAME<span style="white-space:pre-wrap"> </span>= "Debian Server"</font></div>
<div><font face="'courier new', monospace">VCPU<span style="white-space:pre-wrap"> </span>= 1 </font></div><div><font face="'courier new', monospace">MEMORY<span style="white-space:pre-wrap"> </span>= 1024</font></div>
<div><font face="'courier new', monospace">OS<span style="white-space:pre-wrap"> </span>= [ ARCH = "i686" ]</font></div><div><font face="'courier new', monospace">DISK <span style="white-space:pre-wrap"> </span>= [IMAGE="Debian Server"]</font></div>
</div><div><br></div><div>Output:</div><div><div><div style="font-family: 'courier new', monospace; ">oneadmin@custom2:~$ onehost list</div><div style="font-family: 'courier new', monospace; "> ID NAME CLUSTER RVM TCPU FCPU ACPU TMEM FMEM STAT</div>
<div style="font-family: 'courier new', monospace; "> 2 <a href="http://custom7.sns.it">custom7.sns.it</a> default 3 200 200 200 2G 0K on</div><div style="font-family: 'courier new', monospace; ">
1 <a href="http://custom6.sns.it">custom6.sns.it</a> default 0 200 200 200 2G 0K off</div><div style="font-family: 'courier new', monospace; "><div>oneadmin@custom2:~$ onevm list</div>
<div> ID USER NAME STAT CPU MEM HOSTNAME TIME</div><div> 92 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it">custom7.sns.it</a> 00 00:12:53</div><div> 93 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it">custom7.sns.it</a> 00 00:12:46</div>
<div> 94 oneadmin Debian S runn 0 0K <a href="http://custom7.sns.it">custom7.sns.it</a> 00 00:12:46</div><div><br></div></div><div><font class="Apple-style-span" face="arial, helvetica, sans-serif">I verified if the VM were up and running by logging to the console of each one </font><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">of them </span><span class="Apple-style-span" style="font-family: arial, helvetica, sans-serif; ">through vSphere Client and they were all running and declaring an amount of 1GB of total memory on each one of them. Since there is less than 1.2 GB of memory effectively free on a cluster node before the VMs instantiation how can those VMs run consistently? Why OpenNebula schedule those VM on the same machine exceeding even the host resource capacity?</span></div>
</div></div><br><div class="gmail_quote">On Fri, Feb 4, 2011 at 11:04 PM, Ruben S. Montero <span dir="ltr"><<a href="mailto:rubensm@dacya.ucm.es" target="_blank">rubensm@dacya.ucm.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<div><br></div><div>You have to add also de CPU capacity for the VM (apart from the number of virtual cpus CPUs). The CPU value is used at the allocation phase. However you are specifying MEMORY and should be included in the allocated memeory (USED MEMORY in onehost show) So I guess there should be other problem with your template.</div>
<div><br></div><div>Cheers</div><div><br></div><div><font color="#888888">Ruben</font><div><div></div><div><br><br><div class="gmail_quote">On Fri, Feb 4, 2011 at 10:50 AM, Luigi Fortunati <span dir="ltr"><<a href="mailto:luigi.fortunati@gmail.com" target="_blank">luigi.fortunati@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I can post the VM template content on monday. However, as far as I remember, the vm template was really simple:<div><font face="'courier new', monospace">NAME="Debian"</font></div>
<div><font face="'courier new', monospace">VCPU= 2</font></div><div><font face="'courier new', monospace">MEMORY=1024</font></div><div><font face="'courier new', monospace">DISK=[IMAGE="Debian5-i386"]</font></div>
<div><font face="'courier new', monospace">OS=[ARCH=i686]</font></div><div><font face="'courier new', monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">The VMs can boot and run, I can log on console through vSphere Client on the newly created VMs.</font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="arial, helvetica, sans-serif">I noticed that if you don't declare the number on VCPU the VM doesn't get scheduled on a cluster node. This option seems mandatory but I didn't find any mention about it on the documentation.</font></div>
<div><font face="arial, helvetica, sans-serif">Another thing that seems mandatory is declaring the cpu architecture as i686, otherwise OpenNebula will return error when writing the deployment.0 file.</font></div>
<div><div><div></div><div><br><br><div class="gmail_quote">On Thu, Feb 3, 2011 at 5:42 PM, Ruben S. Montero <span dir="ltr"><<a href="mailto:rubensm@dacya.ucm.es" target="_blank">rubensm@dacya.ucm.es</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<div><br></div><div>I am not sure this is related to the VMware monitoring... Can you send the VM Templates?</div><div><br></div><div>Thanks</div><div><br></div><div>Ruben<br><br><div class="gmail_quote"><div><div></div>
<div>On Thu, Feb 3, 2011 at 5:10 PM, Luigi Fortunati <span dir="ltr"><<a href="mailto:luigi.fortunati@gmail.com" target="_blank">luigi.fortunati@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div>Hi,<div>I noticed a serious problem about the usage of VMWare ESXi 4.1 and OpenNebula 2.0.1.</div>
<div>I'm actually using the VMWare driver addon which can be found on the opennebula website (ver. 1.0) and libvirt (ver. 0.8.7).</div>
<div>It happens that OpenNebula can't get information about the usage of resources on the cluster nodes.</div><div>By running 2 VM (each one requires 2 VCPU and 1 GB of memory) and executing some commands I get this output.</div>
<div><br></div><div><div><font face="'courier new', monospace">oneadmin@custom2:~/src$ onehost list</font></div><div><font face="'courier new', monospace"> ID NAME CLUSTER RVM TCPU FCPU ACPU TMEM FMEM STAT</font></div>
<div><font face="'courier new', monospace"> 2 <a href="http://custom7.sns.it" target="_blank">custom7.sns.it</a> default 0 200 200 200 2G 0K off</font></div><div><font face="'courier new', monospace"> 1 <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> default 2 200 200 200 2G 0K on</font></div>
<div><font face="'courier new', monospace">oneadmin@custom2:~/src$ onehost show 1</font></div><div><font face="'courier new', monospace">HOST 1 INFORMATION </font></div>
<div><font face="'courier new', monospace">ID : 1 </font></div><div><font face="'courier new', monospace">NAME : <a href="http://custom6.sns.it" target="_blank">custom6.sns.it</a> </font></div>
<div><font face="'courier new', monospace">CLUSTER : default </font></div><div><font face="'courier new', monospace">STATE : MONITORED </font></div>
<div><font face="'courier new', monospace">IM_MAD : im_vmware </font></div><div><font face="'courier new', monospace">VM_MAD : vmm_vmware </font></div>
<div><font face="'courier new', monospace">TM_MAD : tm_vmware </font></div><div><font face="'courier new', monospace"><br></font></div>
<div><font face="'courier new', monospace">HOST SHARES </font></div><div><font face="'courier new', monospace">MAX MEM : 2096460 </font></div>
<div><font face="'courier new', monospace">USED MEM (REAL) : 0 </font></div><div><font face="'courier new', monospace">USED MEM (ALLOCATED) : 0 </font></div>
<div><font face="'courier new', monospace">MAX CPU : 200 </font></div><div><font face="'courier new', monospace">USED CPU (REAL) : 0 </font></div>
<div><font face="'courier new', monospace">USED CPU (ALLOCATED) : 0 </font></div><div><font face="'courier new', monospace">RUNNING VMS : 2 </font></div>
<div><font face="'courier new', monospace"><br></font></div><div><font face="'courier new', monospace">MONITORING INFORMATION </font></div>
<div><font face="'courier new', monospace">CPUSPEED=1992</font></div><div><font face="'courier new', monospace">HYPERVISOR=vmware</font></div><div><font face="'courier new', monospace">TOTALCPU=200</font></div>
<div><font face="'courier new', monospace">TOTALMEMORY=2096460</font></div></div><div><font face="'courier new', monospace"><br></font></div><div>As you can see OpenNebula is unable to get correct information about the usage of resources on the cluster nodes.</div>
<div>As these informations are used by the VM scheduler, OpenNebula is unable to schedule the VM correctly.</div><div>I tried to create several VM and all of them were placed on the same host even if the latter was unable to satisfy the resource requirements of all the VMs.</div>
<div>I think that this problem is strongly related to libvirt as OpenNebula use it to recover information about hosts and vm.</div><div><br></div><div>Do you get the same behavior? Do you know if there is a way to solve this big issue?</div>
<div><br>-- <br>Luigi Fortunati<br>
</div>
<br></div></div>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Dr. Ruben Santiago Montero<br>Associate Professor (Profesor Titular), Complutense University of Madrid<br><br>URL: <a href="http://dsa-research.org/doku.php?id=people:ruben" target="_blank">http://dsa-research.org/doku.php?id=people:ruben</a><br>
Weblog: <a href="http://blog.dsa-research.org/?author=7" target="_blank">http://blog.dsa-research.org/?author=7</a><br>
</div>
</blockquote></div><br><br clear="all"><br></div></div>-- <br>Luigi Fortunati<br>
</div>
</blockquote></div><br><br clear="all"><br>-- <br>Dr. Ruben Santiago Montero<br>Associate Professor (Profesor Titular), Complutense University of Madrid<br><br>URL: <a href="http://dsa-research.org/doku.php?id=people:ruben" target="_blank">http://dsa-research.org/doku.php?id=people:ruben</a><br>
Weblog: <a href="http://blog.dsa-research.org/?author=7" target="_blank">http://blog.dsa-research.org/?author=7</a><br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>Luigi Fortunati<br>
</div></div>