Hi Michael,<div><br></div><div>Thank you for sharing your code and for your great feedback.</div><div>I have created a wiki page [1] to give this thread more visibility.</div><div><br></div><div>A quick note for anyone interested in this migration: Michael's code can be merged into 3.3.0_to_3.3.80.rb or 3.3.80_to_3.4.0.rb to skip the <span style>db_versioning table edition.</span></div>
<div><br></div><div>Cheers</div><div><br></div><div>[1] <a href="http://wiki.opennebula.org/upgrade34" target="_blank">http://wiki.opennebula.org/upgrade34</a><br clear="all">--<br>
Carlos Martín, MSc<br>Project Engineer<br>OpenNebula - The Open-source Solution for Data Center Virtualization<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="mailto:cmartin@opennebula.org" style="color:rgb(42,93,176)" target="_blank"></a></span></div>
<br>
<br><br><div class="gmail_quote">On Fri, Apr 20, 2012 at 7:49 AM, Michael Kutzner <span dir="ltr"><<a href="mailto:michael.kutzner@virtion.de" target="_blank">michael.kutzner@virtion.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div><div>Hi,</div><div><br></div><div>the following is a longer email, but I believe there are others also having the need to make a "smooth" upgrade from 3.2.x to 3.4</div><div>
<br></div><div>Thanx again to Carlos to show me the right way, so how did we do the Migration of 3.2.1 to 3.4 having certain active VMs and not the chance to shut them down.</div><div>Some background for the steps described below:</div>
<div>Our old setup was:</div><div>- ssh as transfer manager (a little bit modified)</div><div>- VM_DIR was /one/vm in 3.2</div><div>- lots of persistent images, no images marked with SAVE_AS</div><div>- path to image repository: /one/var/images</div>
<div><br></div><div>New setup:</div><div>- ssh transfer manager</div><div>- filesystem datastore 1 (/one/var/datastores/1)</div><div>- system datastore 0 (/one/var/datastores/0)</div><div><br></div><div>The migration steps:</div>
<div>1) Normal upgrade procedure (installation of one 3.4 without onedb upgrade yet)</div><div>2) as describe by Carlos I commented out lines 36 to 60 in</div><div> /usr/lib/one/ruby/onedb/3.3.0_to_3.3.80.rb</div><div>
(line 35 does not have to be commented out - "header_done=false")</div><div> onedb upgrade run, working fine</div><div>3) On each node create a link for each active vm from</div><div> /one/var/datastores/1/<VM_ID> -> /one/vm/<VM_ID>/images</div>
<div><div>--------- <script>---------</div><div>#!/bin/sh</div><div>#</div><div>DATASTORE_LOCATION=/one/var/datastores</div><div>SYSTEM_STORE=$DATASTORE_LOCATION/0</div><div>IMAGE_STORE=$DATASTORE_LOCATION/1</div><div>
VM_DIR=/one/vm</div><div><br></div><div># Create Datastore Directory</div><div>test -d $DATASTORE_LOCATION || mkdir $DATASTORE_LOCATION</div><div># Create System Datastore</div><div>test -d $SYSTEM_STORE || (echo "creating vm dir $SYSTEM_STORE"; mkdir $SYSTEM_STORE)</div>
<div>#</div><div># In our setup the datastore 1 is available on each host, but was mounted to /one/images</div><div>#</div><div>test -L $IMAGE_STORE || (echo "linking $IMAGE_STORE"; ln -s /one/images $IMAGE_STORE )</div>
<div><br></div><div><br></div><div>for i in $VM_DIR/*; do</div><div> ID=`basename $i`</div><div> if [ -d $i/images -a ! -L $SYSTEM_STORE/$ID -a ! -d $SYSTEM_STORE/$ID ]; then</div><div> echo "linking running vm $ID to new system datastore"</div>
<div> ln -s $i/images $SYSTEM_STORE/$ID</div><div> fi</div><div>done</div><div>------------------------------</div><div><br></div><div>After this step ONE was able to monitor again the running VMs, but as I noticed at the first test, it was not to shutdown the vm and migrate the images back to the image datastore 1.</div>
<div>What was the problem?</div><div>Well the information of the TM_MAD and the datastores was missing in the persistent images - this information is stored in the the XML body in the one database, table vm_pool.</div><div>
<DATASTORE>default</DATASTORE></div><div> <DATASTORE_ID>1</DATASTORE></div><div> <TM_MAD>ssh</TM_MAD></div><div>This information is needed to use the correct transfer manager for the persistent image migration.</div>
<div>(This creates an error like:</div><div> Error executing image transfer script: Transfer Driver 'hostname:/one/var/datastores/0/1322/disk.0' not available)</div><div><br></div><div>So we had to add the needed information to each persistent disk on each active vm.</div>
<div><br></div><div>Diving into ruby for the first time in my life I decided to do this creating a kind of db update script in for "onedb" way as this brings out of the box every thing needed (mainly just some data changes in the database) and was the fastest way for me.</div>
<div>What had to be done?</div><div>I created an update script as attached to this email, which does the following:</div><div>- Adds the above mentioned informations to each image, if it is persistent</div><div> and not a vm started in the new way (starting at line 46).</div>
<div>- Changes the SOURCE of the images to the correct new location.</div></div></div><div><br></div><div><div>This worked fine for us using again the normal "onedb upgrade", but to start oned again - now db_version is set to 3.4.0.5 I had to delete the last entry from the table db_versioning for this version.</div>
<div>(just do a select * from db_versioning, note the oid and delete the row with the oid having the </div><div> entry for version 3.4.0.5)</div><div><br></div><div>After having done this ONE was running fine and shutting down some "older" test vms worked also fine - migrating back the persistent images to the datastore 1.</div>
<div><br></div><div>Beware that when shutting down the "old" vms, only the links on the nodes are deleted, not the images themselves. This may be done manually or by a small extension to the delete script of the ssh transfer driver.</div>
<div><br></div><div>Perhaps this mail gives some info to people having the same need for the upgrade.</div><div><br></div><div><br></div><div>Best, Michael</div><div><br></div><div>PS: I like the new version, great work</div>
</div><div><br></div><div></div><span style="white-space:pre-wrap">[siehe angehängte Datei: 3.4.0_to_3.4.0.5.rb]</span><div style="word-wrap:break-word"><div></div><div><div><br></div><div><br></div><div>Am 12.04.2012 um 16:26 schrieb Carlos Martín Sánchez:</div>
<br><blockquote type="cite"><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">Hi Michael,</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
The reason why VMs have to be shut down is because the storage model has changed quite a bit in this last version, and we couldn't come up with a migration process that felt solid enough for any deployment.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">But, since you said you are ready for lots of manual work, let's try this:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">First, edit /usr/lib/one/ruby/onedb/3.3.0_to_3.3.80.rb and comment out lines 35 to 60 [1] to disable the VM status check.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
Then upgrade the opennebula DB with the onedb command [2]</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
Before you start opennebula again, you need to adapt the previous VM storage directories to the new 3.4 hierarchy. Assuming you don't have VM_DIR nor DATASTORE_LOCATION defined in oned.conf, this is how the VM directories look in both versions:</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">3.2:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
/var/lib/one/<vm_id>/images/disk.i</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">/var/lib/images/abcde</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">3.4:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">/var/lib/one/datastores/0/<vm_id>/disk.i</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">/var/lib/one/datastores/1/abcde</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
So what you need to do in the front-end *and each host* is to individually link each existing VM dir:</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
ln -s /var/lib/one/<vm_id>/images /var/lib/one/datastores/0/<vm_id></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
Let me stress that this must be done in all the hosts.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
Please take into account that this is just the basic idea, and that *we haven't tested this*. You may run into problems depending on your infrastructure configuration. For instance, if you were using the tm_shared drivers, VMs containing persistent Images will have links like this one:</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">/var/lib/one/7/images/disk.1 -> /var/lib/images/qwerty</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">If you move the image files to their new destination in /var/lib/one/datastores/1, like the documentation says, you will break those VM disk links.</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
Another problem that may be important to you is that VMs over which you executed saveas [3] won't be able to save the changes back to the locked Image once the VM is shut down. In the best scenario, you will find the files inside /var/lib/one/<vm_id>/images/, but I'm not sure about this, you may lose your changes...</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">You can locate these VMs looking for VM/DISK/SAVE_AS, and the Images all have "-" as the value of IMAGE/SOURCE.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">As I said, this is not a tested or recommended procedure.</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
Good luck,</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">Carlos</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">
[1] <a href="http://dev.opennebula.org/projects/opennebula/repository/revisions/one-3.4/entry/src/onedb/3.3.0_to_3.3.80.rb" style="color:rgb(17,85,204)" target="_blank">http://dev.opennebula.org/projects/opennebula/repository/revisions/one-3.4/entry/src/onedb/3.3.0_to_3.3.80.rb</a></div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13px">[2] <a href="http://opennebula.org/documentation:rel3.4:upgrade" style="color:rgb(17,85,204)" target="_blank">http://opennebula.org/documentation:rel3.4:upgrade</a></div>
<div><font color="#222222" face="arial, sans-serif">[3] <a href="http://opennebula.org/documentation:archives:rel3.2:img_guide#save_changes" target="_blank">http://opennebula.org/documentation:archives:rel3.2:img_guide#save_changes</a></font></div>
<div><font color="#222222" face="arial, sans-serif"><br></font></div></div>--<br>Carlos Martín, MSc<br>Project Engineer<br>OpenNebula - The Open-source Solution for Data Center Virtualization<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org/" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="mailto:cmartin@opennebula.org" style="color:rgb(42,93,176)" target="_blank"></a></span></div>
<br>
<br><br><div class="gmail_quote">On Thu, Apr 12, 2012 at 2:41 PM, Michael Kutzner <span dir="ltr"><<a href="mailto:michael.kutzner@virtion.de" target="_blank">michael.kutzner@virtion.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
first of all congratulations to the final 3.4 and many thanx for the hard work!<br>
<br>
I just wanted to upgrade to 3.4 and started reading.<br>
I noticed reading the upgrade guide that all active VMs have to be shut down.<br>
<br>
Is there any chance to upgrade without powering down all VMs? Even with lots of<br>
manual work?<br>
<br>
<br>
Many thanx,<br>
Michael<br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</blockquote></div><br>
</blockquote></div><br><div>
<div style="word-wrap:break-word"><div style="word-wrap:break-word"><div style="word-wrap:break-word"><div><div><div><br></div></div></div></div></div></div><br>
</div>
<br></div></div><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br></div>