[one-users] Image upload within the core

[Tino Vazquez]

Hi Csom,

Our approach to image upload is:

* Private clouds: The ImagePool will handle the metadata of the
images. The physical files would be handled by the CLI, transferring
files using the unix filesystem copy command. This will offer the
possibility to update the file and eventually delete it.

* Public clouds: Current repository manager in the OCCI and EC2
servers will be swapped with the ImagePool. Uploading of images will
be performed using pure http as currently.

We appreciate any feedback, comment on this.

Best regards,

-Tino

--
Constantino Vázquez, Grid & Virtualization Technology
Engineer/Researcher: http://www.dsa-research.org/tinova
DSA Research Group: http://dsa-research.org
Globus GridWay Metascheduler: http://www.GridWay.org
OpenNebula Virtual Infrastructure Engine: http://www.OpenNebula.org



2010/6/24 Csom Gyula <csom at interface.hu>:
> Hi!
>
> Thanks Todd for your response! It was really helpful... maybe not directly, but it gave me a useful tipp to look for cloud-oriented data transfer solutions.
>
> I Googled the topic but so far haven't found much. After all here's the list:
>
> * There's GridFTP you proposed.
> * CDMI [1] which is a rather complex standard even donno whether it supports file uploads or not:))
> * UDT [2] which is a brand new technology meanwhile seems to be the fastest solution among data transfer methods (Supercomputing Bandwidth Challenge Winner at 2006, 2008 and 2009).
> * And of course one can always choose well known protocols like scp, sftp, pure http, etc. [3]
>
> Cheers
> Gyula
>
> ---
>
> [1] CDMI: http://cloud-standards.org/wiki/index.php?title=SNIA_Cloud_Data_Management_Interface_%28CDMI%29
> [2] UDT: http://en.wikipedia.org/wiki/UDP-based_Data_Transfer_Protocol, http://udt.sourceforge.net/
> ________________________________________
> Feladó: Todd Deshane [deshantm at gmail.com]
> Küldve: 2010. június 23. 23:53
> Címzett: Csom Gyula
> Másolatot kap: users at lists.opennebula.org
> Tárgy: Re: [one-users] Image upload within the core
>
> Hi Gyula,
>
> I can't answer the roadmap question, but just to let you know about a
> couple related projects in case you were unaware of them.
>
> For sending large amounts of data (such as disk images), Nimbus [1]
> uses GridFTP [2].
>
> Another really promising project for creating base images and
> filesystem stacks is a project called Stacklet [3,4].
>
> Hope that helps.
>
> Thanks,
> Todd
>
> [1] http://www.nimbusproject.org/
> [2] http://www.globus.org/toolkit/data/gridftp/
> [3] http://stacklet.com/
> [4] http://bitbucket.org/stacklet/stacklet/
>
> On Wed, Jun 23, 2010 at 3:09 PM, Csom Gyula <csom at interface.hu> wrote:
>> Hi!
>>
>>
>> Do you have plans to support image uploads? Is it on your 1.6 roadmap? I ask this
>> in order to coordinate our (extension) development with your roadmap. Some
>> background:
>>
>>
>> Currently we are in the procces to specify our golden image management service, eg:
>>
>> what features to provide exactly ? how to implement it? in a maintanable manner (eg.
>>
>> starting with ONE v.1.4 then smoothly migrate to v.1.6)?
>>
>>
>> We've found that one of the biggest challenges is the image upload functionality.
>>
>> A clean solution would implement the upload service within the core. Meanwhile
>> XML-RPC used by the RequestManager is inappropriate for large file uploads.
>> XML-RPC provides nothing but base64 types for binary data, meanwhile base64
>> coding-decoding of large files is a pain... Also xmlrpc-c used by ONE seems to
>> represent binary data in byte arrays in memory which is unacceptable for large,
>> multi-GB files...
>>
>>
>> So if we decide to implement the upload service we will likely choose an alternative
>> approach, eg. a separate channel for text data (XML-RPC) and another one for large
>> files (scp, http whatever seems most reasonable). Meanwhile we'd like to adhere to
>> the ONE mainline as much as possible.
>>
>>
>> BTW: We are aware of the upcomming image repository feature [1]. To our
>>
>> understanding (after a quick code walk through) it provides image metadata-only
>>
>> service.
>> .
>>
>> Cheers,
>> Gyula
>>
>>
>> [1] http://dev.opennebula.org/issues/200
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>
>
>
> --
> Todd Deshane
> http://todddeshane.net
> http://runningxen.com
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>