[one-users] OpenNebula 3.8.3 and VMWare ESXi 5.0: internal error HTTP response code 403 for upload

chenxiang at aquala-tech.com chenxiang at aquala-tech.com
Wed Apr 24 03:28:41 PDT 2013 

The disk.vmdk problem seems to be associated with the DS_MAD setting. I
created two datastores to test this, the first one with DS_MAD="vmware"
and the second one with DS_MAD="fs". When I imported the images from
marketplace, in the first datastore I have disk.vmdk, and in the other one
I have ttylinux.vmdk.

When I encountered the problem in this thread, I was importing the
ttylinux image into the default datastore, which had DS_MAD="fs". So, that
caused the problem.

A lot of things to learn. Thank you so much for your assistance. You are
so helpful for me.

Chen Xiang


> On Wed, Apr 24, 2013 at 7:39 AM,  <chenxiang at aquala-tech.com> wrote:
>> I double checked the VNC issue. I did set the VNC stuff according to the
>> instructions, but that setting got lost across reboots, so I was not
>> able
>> to connect to VNC.
>>
>> I searched on the web, and found a solution. That is to copy the
>> modified
>> version of service.xml to the localdisk of ESXi as a backup, and then
>> modified /etc/rc.local to recover that file from the backup version, and
>> to refresh the firewall settings. (I need to do the same for the
>> oneadmin
>> SSH key, otherwise the front end won't be able to connect to the ESXi
>> box
>> when the box reboots.) Then VNC worked.
>
> Glad you got it working. A similar approach is defined here [1]
> (search for "persistency")
>
> [1] http://opennebula.org/documentation:rel3.8:evmwareg
>
>>
>> Regarding the disk.vmdk. I checked into the oned.log. No I did not find
>> the error message you mentioned.
>
> Can you send us the relevant files in oned.log at the time of
> registering a VMware disk folder _without_ the manual renaming to
> disk.vmdk?
>
> Thanks a lot for your feedback,
>
> -T
>
>>
>> Chen Xiang
>>
>>
>>> Hi,
>>>
>>> comments inline,
>>>
>>> On Tue, Apr 23, 2013 at 8:54 AM,  <chenxiang at aquala-tech.com> wrote:
>>>> Now I am able to get VM's running with ESXi nodes. I would like to
>>>> share
>>>> what I did to make things work, and would like you guys to help me
>>>> further.
>>>>
>>>> From Marketplace I imported the ttylinux-vmware image to local
>>>> infrastructure. When the images becomes READY, login to the front end
>>>> server and browse to var/datastores/1 and further browse to the
>>>> directory
>>>> holding that particular image. Below was what I had when things did
>>>> not
>>>> work:
>>>>
>>>> oneadmin at opennebula:~/var/datastores/1/10fc21f21a452add3838d76d63052457$
>>>> ls -l
>>>> total 104864
>>>> -rw------- 1 oneadmin cloud 107374080 Jul  3  2012 ttylinux-flat.vmdk
>>>> -rw------- 1 oneadmin cloud       509 Jul  3  2012 ttylinux.vmdk
>>>>
>>>> Base on the error message I got from VM log, I decided to copy
>>>> ttylinux.vmdk to disk.vmdk. So I did that and then I had the
>>>> following:
>>>>
>>>> oneadmin at opennebula:~/var/datastores/1/10fc21f21a452add3838d76d63052457$
>>>> cp ttylinux.vmdk disk.vmdk
>>>> oneadmin at opennebula:~/var/datastores/1/10fc21f21a452add3838d76d63052457$
>>>> ls -l
>>>> total 104868
>>>> -rw------- 1 oneadmin cloud       509 Apr 23 14:33 disk.vmdk
>>>> -rw------- 1 oneadmin cloud 107374080 Jul  3  2012 ttylinux-flat.vmdk
>>>> -rw------- 1 oneadmin cloud       509 Jul  3  2012 ttylinux.vmdk
>>>>
>>>> Now I went back to the same template, and instantiated a VM instance,
>>>> now
>>>> it worked.
>>>
>>> The vmware/cp script should automatically rename the file. Can you see
>>> any line in /var/log/one/oned.log similar to:
>>>
>>> --
>>> Error renaming disk file $BASE_DISK_FILE to disk.vmdk
>>> --
>>>
>>>
>>>>
>>>> Now I have a new problem. This front-end has been tested for KVM with
>>>> hundreds of VM instances before I tried ESXi. Now I am trying VM-ID
>>>> above
>>>> 100. Now the VM instances were running, but I was not able to connect
>>>> to
>>>> the VM console via VNC from SunStone. What should I do? (VM Console in
>>>> vSphere Client still works.)
>>>
>>>
>>> Have you configured the ESX host to allow VNC connections? See
>>> http://opennebula.org/documentation:rel3.8:evmwareg#vnc
>>>
>>> Regards,
>>>
>>> -Tino
>>>
>>>>
>>>> Thanks a lot for your assistance.
>>>>
>>>> Chen Xiang
>>>>
>>>>
>>>>
>>>>> With the proper NFS settings, I was able to define the domain, but
>>>>> fail
>>>>> to
>>>>> create the VM. The error message says "File [0] 120/disk.0/disk.vmdk
>>>>> was
>>>>> not found".
>>>>>
>>>>> I login to the ESXi box, and verified that I do have /vmfs/volumes/0
>>>>> pointing to the right NFS mount. And inside /vmfs/volumes/0 I do have
>>>>> 120/disk.0/disk.vmdk .
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Below is a copy of the VM log.
>>>>>
>>>>> Tue Apr 23 13:12:58 2013 [DiM][I]: New VM state is ACTIVE.
>>>>> Tue Apr 23 13:12:58 2013 [LCM][I]: New VM state is PROLOG.
>>>>> Tue Apr 23 13:12:58 2013 [VM][I]: Virtual Machine has no context
>>>>> Tue Apr 23 13:13:12 2013 [TM][I]: clone: Cloning
>>>>> /vmfs/volumes/1/10fc21f21a452add3838d76d63052457 in
>>>>> vmware02:/vmfs/volumes/0/120/disk.0
>>>>> Tue Apr 23 13:13:12 2013 [TM][I]: ExitCode: 0
>>>>> Tue Apr 23 13:13:12 2013 [LCM][I]: New VM state is BOOT
>>>>> Tue Apr 23 13:13:12 2013 [VMM][I]: Generating deployment file:
>>>>> /srv/cloud/one/var/vms/120/deployment.0
>>>>> Tue Apr 23 13:13:12 2013 [VMM][I]: ExitCode: 0
>>>>> Tue Apr 23 13:13:12 2013 [VMM][I]: Successfully execute network
>>>>> driver
>>>>> operation: pre.
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]: Command execution fail:
>>>>> /srv/cloud/one/var/remotes/vmm/vmware/deploy
>>>>> /srv/cloud/one/var/vms/120/deployment.0 vmware02 120 vmware02
>>>>> Tue Apr 23 13:13:24 2013 [VMM][D]: deploy: Successfully defined
>>>>> domain
>>>>> one-120.
>>>>> Tue Apr 23 13:13:24 2013 [VMM][E]: deploy: Error executing: virsh -c
>>>>> 'esx://vmware02/?no_verify=1&auto_answer=1' start one-120 err:
>>>>> ExitCode: 1
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]: out:
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]: error: Failed to start domain
>>>>> one-120
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]: error: internal error Could not
>>>>> start
>>>>> domain: FileNotFound - File [0] 120/disk.0/disk.vmdk was not found
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]:
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]: ExitCode: 1
>>>>> Tue Apr 23 13:13:24 2013 [VMM][I]: Failed to execute virtualization
>>>>> driver
>>>>> operation: deploy.
>>>>> Tue Apr 23 13:13:24 2013 [VMM][E]: Error deploying virtual machine
>>>>> Tue Apr 23 13:13:24 2013 [DiM][I]: New VM state is FAILED
>>>>>
>>>>>
>>>>>
>>>>> Chen Xiang
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Please use root_squash instead of no_root_squash
>>>>>>
>>>>>> --
>>>>>> /srv/cloud/one/var/datastores/0
>>>>>> *(rw,sync,no_subtree_check,no_root_squash,anonuid=10000,anongid=10000)
>>>>>> /srv/cloud/one/var/datastores/1
>>>>>> *(rw,sync,no_subtree_check,no_root_squash,anonuid=10000,anongid=10000)
>>>>>> --
>>>>>>
>>>>>> You will need to force the nfs server to re-read the conf file.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> -Tino
>>>>>> --
>>>>>> Constantino Vázquez Blanco, PhD, MSc
>>>>>> Project Engineer
>>>>>> OpenNebula - The Open-Source Solution for Data Center Virtualization
>>>>>> www.OpenNebula.org | @tinova79 | @OpenNebula
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 22, 2013 at 3:46 PM,  <chenxiang at aquala-tech.com> wrote:
>>>>>>> Here is what I got when trying to define the VM via virsh:
>>>>>>>
>>>>>>> oneadmin at opennebula:~/images$ virsh -c
>>>>>>> 'esx://vmware02/?no_verify=1&auto_answer=1'
>>>>>>> Enter username for vmware02 [root]:
>>>>>>> Enter root's password for vmware02:
>>>>>>> Welcome to virsh, the virtualization interactive terminal.
>>>>>>>
>>>>>>> Type:  'help' for help with commands
>>>>>>>        'quit' to quit
>>>>>>>
>>>>>>> virsh # define /srv/cloud/one/var/vms/111/deployment.0
>>>>>>> 2013-04-22 13:24:49.391+0000: 17332: info : libvirt version: 0.9.10
>>>>>>> 2013-04-22 13:24:49.391+0000: 17332: warning : virVMXFormatVNC:3224
>>>>>>> :
>>>>>>> VNC
>>>>>>> port 6011 it out of [5900..5964] range
>>>>>>> error: Failed to define domain from
>>>>>>> /srv/cloud/one/var/vms/111/deployment.0
>>>>>>> error: internal error HTTP response code 403 for upload to
>>>>>>> 'https://vmware02:443/folder/111%2fdisk%2e0/one%2d111.vmx?dcPath=ha%2ddatacenter&dsName=0'
>>>>>>>
>>>>>>> virsh # exit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am running OpenNebula 3.8.3 on Ubuntu 12.04 (front end) with
>>>>>>>> VMWare
>>>>>>>> ESXi
>>>>>>>> 5.0 (node). I am able to create a VMWare node in SunStone,
>>>>>>>> register
>>>>>>>> the
>>>>>>>> ttylinux images (ttylinux.vmdk.tar download from C12G.com), but
>>>>>>>> failed
>>>>>>>> to
>>>>>>>> instantiate a VM.
>>>>>>>>
>>>>>>>> Here is what I have as the NFS exports:
>>>>>>>>
>>>>>>>> /srv/cloud/one/var/datastores/0
>>>>>>>> *(rw,sync,no_subtree_check,no_root_squash,anonuid=10000,anongid=10000)
>>>>>>>> /srv/cloud/one/var/datastores/1
>>>>>>>> *(rw,sync,no_subtree_check,no_root_squash,anonuid=10000,anongid=10000)
>>>>>>>>
>>>>>>>> One the ESXi node I mounted the NFS exports to /vmfs/volumes/0 and
>>>>>>>> /vmfs/volumes/1 respectively.
>>>>>>>>
>>>>>>>> Yes I configured libvirt-0.9.10 and make it worked with VMWare,
>>>>>>>> verified
>>>>>>>> by command lines such as the following command (where vmware02 is
>>>>>>>> my
>>>>>>>> ESXi
>>>>>>>> hostname):
>>>>>>>>
>>>>>>>> virsh -c 'esx://vmware02/?no_verify=1&auto_answer=1'
>>>>>>>>
>>>>>>>> Below is my VM template:
>>>>>>>>
>>>>>>>> CPU="1"
>>>>>>>> DISK=[
>>>>>>>>   IMAGE="tty_vmdk",
>>>>>>>>   IMAGE_UNAME="oneadmin" ]
>>>>>>>> GRAPHICS=[
>>>>>>>>   LISTEN="0.0.0.0",
>>>>>>>>   TYPE="vnc" ]
>>>>>>>> MEMORY="512"
>>>>>>>> NAME="ttylinux"
>>>>>>>>
>>>>>>>> Below is what I got when trying to instantiate a VM:
>>>>>>>>
>>>>>>>>
>>>>>>>> Mon Apr 22 21:27:11 2013 [DiM][I]: New VM state is ACTIVE.
>>>>>>>> Mon Apr 22 21:27:11 2013 [LCM][I]: New VM state is PROLOG.
>>>>>>>> Mon Apr 22 21:27:11 2013 [VM][I]: Virtual Machine has no context
>>>>>>>> Mon Apr 22 21:27:17 2013 [TM][I]: clone: Cloning
>>>>>>>> /vmfs/volumes/1/43352fb75cee9bbc1da3c1e7ff474e26 in
>>>>>>>> vmware02:/vmfs/volumes/0/112/disk.0
>>>>>>>> Mon Apr 22 21:27:17 2013 [TM][I]: ExitCode: 0
>>>>>>>> Mon Apr 22 21:27:17 2013 [LCM][I]: New VM state is BOOT
>>>>>>>> Mon Apr 22 21:27:17 2013 [VMM][I]: Generating deployment file:
>>>>>>>> /srv/cloud/one/var/vms/112/deployment.0
>>>>>>>> Mon Apr 22 21:27:17 2013 [VMM][I]: ExitCode: 0
>>>>>>>> Mon Apr 22 21:27:17 2013 [VMM][I]: Successfully execute network
>>>>>>>> driver
>>>>>>>> operation: pre.
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: Command execution fail:
>>>>>>>> /srv/cloud/one/var/remotes/vmm/vmware/deploy
>>>>>>>> /srv/cloud/one/var/vms/112/deployment.0 vmware02 112 vmware02
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][E]: deploy: Error executing: virsh
>>>>>>>> -c
>>>>>>>> 'esx://vmware02/?no_verify=1&auto_answer=1' define
>>>>>>>> /srv/cloud/one/var/vms/112/deployment.0 err: ExitCode: 1
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: out:
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: 2013-04-22 13:27:21.858+0000:
>>>>>>>> 17586:
>>>>>>>> info : libvirt version: 0.9.10
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: 2013-04-22 13:27:21.858+0000:
>>>>>>>> 17586:
>>>>>>>> warning : virVMXFormatVNC:3224 : VNC port 6012 it out of
>>>>>>>> [5900..5964]
>>>>>>>> range
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: error: Failed to define domain
>>>>>>>> from
>>>>>>>> /srv/cloud/one/var/vms/112/deployment.0
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: error: internal error HTTP
>>>>>>>> response
>>>>>>>> code 403 for upload to
>>>>>>>> 'https://vmware02:443/folder/112%2fdisk%2e0/one%2d112.vmx?dcPath=ha%2ddatacenter&dsName=0'
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]:
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: ExitCode: 255
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][I]: Failed to execute
>>>>>>>> virtualization
>>>>>>>> driver
>>>>>>>> operation: deploy.
>>>>>>>> Mon Apr 22 21:27:22 2013 [VMM][E]: Error deploying virtual machine
>>>>>>>> Mon Apr 22 21:27:22 2013 [DiM][I]: New VM state is FAILED
>>>>>>>>
>>>>>>>> What might be wrong? This looks like that I did not have write
>>>>>>>> access
>>>>>>>> to
>>>>>>>> some place. So I tried both the oneadmin and root acount in
>>>>>>>> etc/vmwarerc,
>>>>>>>> with not much luck.
>>>>>>>>
>>>>>>>> On the front end the oneadmin user belongs to the following
>>>>>>>> groups:
>>>>>>>> cloud,
>>>>>>>> adm, sudo, libvirtd.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Chen Xiang
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at lists.opennebula.org
>>>>>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at lists.opennebula.org
>>>>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opennebula.org
>>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opennebula.org
>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>

More information about the Users mailing list