[one-users] Image upload within the core

Thu Jun 24 07:52:01 PDT 2010

Hi Tino!

Thanks for your response! I've got the idea:) Some questions regarding the details:

* Private cloud:
1. Will remote terminals be supported for uploads? scp or such?

2. Will metadata and file upload happen in one "transaction" from the CLI perspective? 
or in separate transactions? Eg.: Will the CLI offer a single command for image creation? 
a command that behind the scenes does the image upload (through scp or such) and 
do the metadata creation. Or will the CLI provide 2 separate commands instead: one for 
upload and one for metadata creation?

* Public and private cloud integration:
3. Similar question but now applied to the OCA API (since OCCI and EC2 servers seem to 
use OCA for backend access): will the OCA API offer a single method for image creation? 
or will it provide separate commands for upload and metadata instead?

Cheers,
Gyula
________________________________________
Feladó: tinova79 at gmail.com [tinova79 at gmail.com] ; meghatalmazó: Tino Vazquez [tinova at fdi.ucm.es]
Küldve: 2010. június 24. 16:13
Címzett: Csom Gyula
Másolatot kap: deshantm at gmail.com; users at lists.opennebula.org
Tárgy: Re: [one-users] Image upload within the core

Hi Csom,

Our approach to image upload is:

* Private clouds: The ImagePool will handle the metadata of the
images. The physical files would be handled by the CLI, transferring
files using the unix filesystem copy command. This will offer the
possibility to update the file and eventually delete it.

* Public clouds: Current repository manager in the OCCI and EC2
servers will be swapped with the ImagePool. Uploading of images will
be performed using pure http as currently.

We appreciate any feedback, comment on this.

Best regards,

-Tino

--
Constantino Vázquez, Grid & Virtualization Technology
Engineer/Researcher: http://www.dsa-research.org/tinova
DSA Research Group: http://dsa-research.org
Globus GridWay Metascheduler: http://www.GridWay.org
OpenNebula Virtual Infrastructure Engine: http://www.OpenNebula.org

2010/6/24 Csom Gyula <csom at interface.hu>:
> Hi!
>
> Thanks Todd for your response! It was really helpful... maybe not directly, but it gave me a useful tipp to look for cloud-oriented data transfer solutions.
>
> I Googled the topic but so far haven't found much. After all here's the list:
>
> * There's GridFTP you proposed.
> * CDMI [1] which is a rather complex standard even donno whether it supports file uploads or not:))
> * UDT [2] which is a brand new technology meanwhile seems to be the fastest solution among data transfer methods (Supercomputing Bandwidth Challenge Winner at 2006, 2008 and 2009).
> * And of course one can always choose well known protocols like scp, sftp, pure http, etc. [3]
>
> Cheers
> Gyula
>
> ---
>
> [1] CDMI: http://cloud-standards.org/wiki/index.php?title=SNIA_Cloud_Data_Management_Interface_%28CDMI%29
> [2] UDT: http://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol, http://udt.sourceforge.net/
> ________________________________________
> Feladó: Todd Deshane [deshantm at gmail.com]
> Küldve: 2010. június 23. 23:53
> Címzett: Csom Gyula
> Másolatot kap: users at lists.opennebula.org
> Tárgy: Re: [one-users] Image upload within the core
>
> Hi Gyula,
>
> I can't answer the roadmap question, but just to let you know about a
> couple related projects in case you were unaware of them.
>
> For sending large amounts of data (such as disk images), Nimbus [1]
> uses GridFTP [2].
>
> Another really promising project for creating base images and
> filesystem stacks is a project called Stacklet [3,4].
>
> Hope that helps.
>
> Thanks,
> Todd
>
> [1] http://www.nimbusproject.org/
> [2] http://www.globus.org/toolkit/data/gridftp/
> [3] http://stacklet.com/
> [4] http://bitbucket.org/stacklet/stacklet/
>
> On Wed, Jun 23, 2010 at 3:09 PM, Csom Gyula <csom at interface.hu> wrote:
>> Hi!
>>
>>
>> Do you have plans to support image uploads? Is it on your 1.6 roadmap? I ask this
>> in order to coordinate our (extension) development with your roadmap. Some
>> background:
>>
>>
>> Currently we are in the procces to specify our golden image management service, eg:
>>
>> what features to provide exactly ? how to implement it? in a maintanable manner (eg.
>>
>> starting with ONE v.1.4 then smoothly migrate to v.1.6)?
>>
>>
>> We've found that one of the biggest challenges is the image upload functionality.
>>
>> A clean solution would implement the upload service within the core. Meanwhile
>> XML-RPC used by the RequestManager is inappropriate for large file uploads.
>> XML-RPC provides nothing but base64 types for binary data, meanwhile base64
>> coding-decoding of large files is a pain... Also xmlrpc-c used by ONE seems to
>> represent binary data in byte arrays in memory which is unacceptable for large,
>> multi-GB files...
>>
>>
>> So if we decide to implement the upload service we will likely choose an alternative
>> approach, eg. a separate channel for text data (XML-RPC) and another one for large
>> files (scp, http whatever seems most reasonable). Meanwhile we'd like to adhere to
>> the ONE mainline as much as possible.
>>
>>
>> BTW: We are aware of the upcomming image repository feature [1]. To our
>>
>> understanding (after a quick code walk through) it provides image metadata-only
>>
>> service.
>> .
>>
>> Cheers,
>> Gyula
>>
>>
>> [1] http://dev.opennebula.org/issues/200
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>
>
>
> --
> Todd Deshane
> http://todddeshane.net
> http://runningxen.com
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>