How to upload big files to a rails application?

I have a ruby application that's used by users via a rails web app.

The use case that i need to solve is: each user needs to compile create an entity on database via the web page and put a really big attachment (can be 100gigs or more too) related to the entity.

Now this is too big to be handled by normal upload plugins, i am searching for a way to solve this problem, my ideas are: - delegate the upload via http to backgroundrb - develop a GTK gui and ship it to user to do the upload

any other hint or ideas?


-------------Problems Reply------------

If it's >100GB, you should probably be using an upload manager of some kind. If your network speed is fast enough, you could just build a trusted applet to communicate with the server. The trusted applet would have full file permission, so it would work just like a standalone application. In this manner, you could avoid needing to distribute a separate utility for file transfer.

Alternatively, you could generate some token on the web, and then let users use a standalone application with the given token to upload their file.

These guys have some good info on uploading large files. Maybe this will help:

They kinda go through their process of solving their problem and the steps they took along the way. Worth a peek.

Uploading a 100GB file through your webapp will not be very good. That upload is going to occupy your web server for the time it takes to upload the file making it inaccessible to others.

I'd consider using FTP or perhaps even having your users upload directly to S3 instead and then manage the files from there since I assume that you have to store these files somewhere anyways.

Transferring 100GB over a web interface makes me cringe. It would be much better to write/use a standalone application to do the transfer.

If you are already using Ruby, you can write a Ruby-based app that uploads over FTP. For example:

require 'net/ftp''','username','password') {|ftp|
ftp.put 'filename'
if (ftp.last_response != "266 Transfer complete.\n")
puts "Error with FTP upload\nResponse was: #{ftp.last_response}"

I use this code to upload auto-generated data files to another server that archives them. There are several different libraries for building simple user interfaces in Ruby, and all you would need is a simple window where the user can enter their username and password, select the file to upload, and click the "Go" button.

The Ruby SSH libraries make it possible to do a secure file transfer over SFTP (hint: require 'net/sftp'). I haven't used it myself, but the docs make it look as easy as FTP. IIRC, SFTP has native support for resuming interrupted transfers.

You could also use a utility like WinSCP, which is an open-source tool that can upload using FTP, SFTP, or SCP. For non-Windows systems, there's Cyberduck for OS X and Kasablanka or gFTP for Linux.

Category:ruby on rails Views:1 Time:2010-02-16

Related post

  • Upload a file to a Rails Application from a Ruby command-line application 2010-04-19

    I've been busy on a social networking site for developers for a while (like a mix of github and Facebook), and I want to implement a simple system to create repositories, like on github. For this, I want to create a command-line application in Ruby (

  • Alter an uploaded PDF file within a Rails application 2011-10-06

    I am working on a Rails application (Rails 3.1) and have the ability to upload a PDF file. The PDF file can then be downloaded by other users once they have logged in. What I would really like to do is append the username of the current logged in use

  • What is the best way to upload a file to another Rails application? 2009-06-25

    I 've researched and noticed that ActiveResource lack this functionality. So, what is the current state of the art when doing a file upload? One problem with Guillermo's approach is that the request has to be nested, like this: body = { :file => {

  • How do I copy a file from one rails application to aws ec2 instance 2012-03-15

    I want to copy a file from my one rails application to a remote/ another application hosted on an aws ec2 instance. I have my security group TCP ports 21 and 21 open for ssh and sftp. But I'm not able to figure out any way to do it. I've tried scp so

  • How to upload a file from an android application into the tomcat 6 server 2012-04-11

    Help me please I have searched and searched but I have not found an operational code, a code that allows to upload a file from my android application to my Tomcat server 6, I need the code on the client side and server side. thank you very much -----

  • Sharing Uploaded Files between multiple Rails Applications 2009-12-07

    I have multiple applications (an admin application, a "public"/non-admin application and a web service application) that all share a single database. I've gotten the applications to share models and other code where appropriate, so I don't have multi

  • How to upload a file temporarily in Rails 3? 2011-12-01

    I'm creating CSV-upload functionality for a site of mine. I'm looking to upload a file, parse it, and then dispose of it. I know I can upload and save a file using Paperclip, but that seems a bit like overkill. All I need to do is parse the uploaded

  • How do you use curl to upload a file to a rails server? 2012-01-24

    my rails server has an api, so authentication is handled via an URL parameter. Everything in the app's api receives and responds in XML. But how do you use XML to upload a file? Do I need to have a different respond_to do |format| format.SOMETHING in

  • Best Practices of Uploading XML files in a Web-Application 2009-10-30

    What's the best approach (industry standard) when another business needs to upload an XML file into your web-application on a regular basis? The contents of the XML must be stored in a table. And, the contents of the XML file needs to be inserted int

  • When and where do I require files in a rails application? 2010-01-22

    Let's say I have I have the following file in my lib directory of my rails application: #lib/proxy.rb module SomeService class ServiceProxy def do_something end end end If I want to use ServiceProxy in a model, I can use it like so: #app/models/produ

  • Upload Large Files from ASP .Net Application 2010-02-15

    I am building a website where i need a page where user can upload large video files, i have created WCF service with streaming but i am calling that WCF service from Button_Click event of web page. I have used below mentioned article for WCF service

  • Uploading/Downloading files - Ruby On Rails system 2011-02-10

    I'm attempting to create a simple file hosting system using Ruby On Rails - I have a large ammount of the system setup (including the registration of new files, and stuff) however I've realised there is a bit of a problem - I'm unsure how to actually

  • How can I handle user-uploaded photo files for a Rails app (using paperclip) on Heroku? 2012-01-26

    So I'm making a Rails app which allows Users to create Items, and each Item has an image attachment, which is handled through the Paperclip gem. The Paperclip gem, by default, saves photos to the public folder, both in original (and thumbnail) form.

  • setting the filename for a downloaded file in a rails application 2009-03-05

    I have a controller action that allows a user to download a file with an extension of .ppt . It's not really a powerpoint binary, just an xml-ish format that powerpoint can read. the file is downloaded from the show action of a controller called Elem

  • What is the need to use RVM .gems file in a Rails application? 2012-02-08

    There is my question: I wonder why using a RVM .gems (see to know what I'm tlaking about) in a Rails app while we use Gemfile to install our gems in our project? I think that could be useful when deploying a

  • darwin not able to stream uploaded 3gp files from rails 2011-07-21

    I am trying to upload 3gp files into my rails application on ami instance using paperclip. I then move that 3gp file to darwing streaming server folder to stream it. To have a better identification of file names , it has been prepended with some ids'

  • uploading a file to a website with ruby/rails 2008-12-24

    I am building a rails app to test our flagship product (also web based). The problem is that part of the testing requires using the production app's web interface to upload files. So what i need to do is have the rails app upload these files to the p

  • Uploading & Unzipping files to S3 through Rails hosted on Heroku? 2010-08-09

    I'd like to be able to upload a zip file to my Rails application that contains a number of images. Then I'd like Rails to unzip that file and attach the images inside to my Photo's model via Paperclip, so that they are ultimately stored on my Amazon

  • How to find a File path in Rails 2011-05-12

    I'm using the .foreach method from the Ruby CSV library and I need help in finding a path to a file within my rails application. CSV.foreach("path/to/file.csv") do |row| # use row here... end To upload CSV files I'm using the Paperclip gem which has

Copyright (C), All Rights Reserved.

processed in 0.151 (s). 11 q(s)