UploadFS

UploadFS is a package for the Meteor framework that aims to make file uploading easy, fast and configurable. Some important features are supported like the ability to start, stop or even abort a transfer, securing file access, transforming files on writing or reading...

If you want to support this package and feel graceful for all the work, please share this package with the community or feel free to send me pull requests if you want to contribute.

Also I'll be glad to receive donations, whatever you give it will be much appreciated.

Installation

To install the package, execute this command in the root of your project :

meteor add jalik:ufs

If later you want to remove the package :

meteor remove jalik:ufs

Plugins

In this documentation, I am using the UploadFS.store.Local store which saves files on the filesystem. But since the package is modular, you can install other stores or even create your own store.

Testing

You can test the package by downloading and running UFS-Example which is simple demo of UploadFS.

Mobile Testing

In order to test on mobile builds, ROOT_URL and --mobile-server must be set to your computer's local IP address and port:

export ROOT_URL=http://192.168.1.7:3000 && meteor run android-device --mobile-server=http://192.168.1.7:3000

Configuration

You can access and modify settings via UploadFS.config.

1import {UploadFS} from 'meteor/jalik:ufs';
2
3// Set default permissions for all stores (you can later overwrite the default permissions on each store)
4UploadFS.config.defaultStorePermissions = new UploadFS.StorePermissions({
5    insert(userId, doc) {
6        return userId;
7    },
8    update(userId, doc) {
9        return userId === doc.userId;
10    },
11    remove(userId, doc) {
12        return userId === doc.userId;
13    }
14});
15
16// Use HTTPS in URLs
17UploadFS.config.https = true;
18
19// Activate simulation for slowing file reading
20UploadFS.config.simulateReadDelay = 1000; // 1 sec
21
22// Activate simulation for slowing file uploading
23UploadFS.config.simulateUploadSpeed = 128000; // 128kb/s
24
25// Activate simulation for slowing file writing
26UploadFS.config.simulateWriteDelay = 2000; // 2 sec
27
28// This path will be appended to the site URL, be sure to not put a "/" as first character
29// for example, a PNG file with the _id 12345 in the "photos" store will be available via this URL :
30// http://www.yourdomain.com/uploads/photos/12345.png
31UploadFS.config.storesPath = 'uploads';
32
33// Set the temporary directory where uploading files will be saved
34// before sent to the store.
35UploadFS.config.tmpDir = '/tmp/uploads';
36
37// Set the temporary directory permissions.
38UploadFS.config.tmpDirPermissions = '0700';

Creating a Store (server)

Since v0.6.7, you can share your store between client and server or define it on the server only. Before v0.6.7, a store must be available on the client and the server.

A store is the place where your files are saved, it could be your local hard drive or a distant cloud hosting solution. Let say you have a Photos collection which is used to save the files info.

You need to create the store that will will contains the data of the Photos collection. Note that the name of the store must be unique. In the following example we are using a local filesystem store. Each store has its own options, so refer to the store documentation to see available options.

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos'
10});

Filtering uploads (server)

You can set an UploadFS.Filter to the store to define restrictions on file uploads. Filter is tested before inserting a file in the collection. If the file does not match the filter, it won't be inserted and will not be uploaded.

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos',
10    // Apply a filter to restrict file upload
11    filter: new UploadFS.Filter({
12        minSize: 1,
13        maxSize: 1024 * 1000, // 1MB,
14        contentTypes: ['image/*'],
15        extensions: ['jpg', 'png']
16    })
17});

If you need a more advanced filter, you can pass your own method using the onCheck option.

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos',
10    // Apply a filter to restrict file upload
11    filter: new UploadFS.Filter({
12        onCheck: function(file) {
13            if (file.extension !== 'png') {
14                return false;
15            }
16            return true;
17        }
18    })
19});

Transforming files (server)

If you need to modify the file before saving it to the store, you can to use the transformWrite option. If you want to modify the file before returning it (for display), then use the transformRead option. A common use is to resize/compress images to optimize the uploaded files.

NOTE: Do not forget to install the required libs on your system with NPM (GM, ImageMagick, GraphicsMagicK or whatever you are using).

1import gm from 'gm';
2import {Mongo} from 'meteor/mongo';
3import {UploadFS} from 'meteor/jalik:ufs';
4
5const Photos = new Mongo.Collection('photos');
6
7const PhotoStore = new UploadFS.store.Local({
8    collection: Photos,
9    name: 'photos',
10    path: '/uploads/photos',
11    // Transform file when reading
12    transformRead(from, to, fileId, file, request) {
13        from.pipe(to); // this returns the raw data
14    },
15    // Transform file when writing
16    transformWrite(from, to, fileId, file) {
17        let gm = Npm.require('gm');
18        if (gm) {
19            gm(from)
20                .resize(400, 400)
21                .gravity('Center')
22                .extent(400, 400)
23                .quality(75)
24                .stream().pipe(to);
25        } else {
26            console.error("gm is not available", file);
27        }
28    }
29});

Copying files (since v0.3.6) (server)

You can copy files to other stores on the fly, it could be for backup or just to have alternative versions of the same file (eg: thumbnails). To copy files that are saved in a store, use the copyTo option, you just need to pass an array of stores to copy to.

1import gm from 'gm';
2import {Mongo} from 'meteor/mongo';
3import {UploadFS} from 'meteor/jalik:ufs';
4
5const Files = new Mongo.Collection('files');
6const Thumbnails128 = new Mongo.Collection('thumbnails-128');
7const Thumbnails64 = new Mongo.Collection('thumbnails-64');
8
9const Thumbnail128Store = new UploadFS.store.Local({
10    collection: Thumbnails128,
11    name: 'thumbnails-128',
12    path: '/uploads/thumbsnails/128x128',
13    transformWrite: function(readStream, writeStream, fileId, file) {
14        let gm = Npm.require('gm');
15        if (gm) {
16            gm(from)
17                .resize(128, 128)
18                .gravity('Center')
19                .extent(128, 128)
20                .quality(75)
21                .stream().pipe(to);
22        } else {
23            console.error("gm is not available", file);
24        }
25    }
26});
27
28const Thumbnail64Store = new UploadFS.store.Local({
29    collection: Thumbnails64,
30    name: 'thumbnails-64',
31    path: '/uploads/thumbsnails/64x64',
32    transformWrite: function(readStream, writeStream, fileId, file) {
33        let gm = Npm.require('gm');
34        if (gm) {
35            gm(from)
36                .resize(64, 64)
37                .gravity('Center')
38                .extent(64, 64)
39                .quality(75)
40                .stream().pipe(to);
41        } else {
42            console.error("gm is not available", file);
43        }
44    }
45});
46
47const FileStore = new UploadFS.store.Local({
48    collection: Files,
49    name: 'files',
50    path: '/uploads/files',
51    copyTo: [
52        Thumbnail128Store,
53        Thumbnail64Store
54    ]
55});

You can also manually copy a file to another store by using the copy() method.

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Backups = new Mongo.Collection('backups');
5const Photos = new Mongo.Collection('photos');
6
7const PhotoStore = new UploadFS.store.Local({
8    collection: Photos,
9    name: 'photos',
10    path: '/uploads/photos'
11});
12
13const BackupStore = new UploadFS.store.Local({
14    collection: Backups,
15    name: 'backups',
16    path: '/backups'
17});
18
19PhotoStore.copy(fileId, BackupStore, function(err, copyId, copyFile) {
20    !err && console.log(fileId + ' has been copied as ' + copyId);
21});

All copies contain 2 fields that references the original file, originalId and originalStore. So if you want to display a thumbnail instead of the original file you could do like this :

1<template name="files">
2    {{#each files}}
3        <img src="{{thumb.url}}">
4    {{/each}}
5</template>

1Template.files.helpers({
2    files: function() {
3        return Files.find();
4    },
5    thumb: function() {
6        return Thumbnails128.findOne({originalId: this._id});
7    }
8});

Or you can save the thumbnails URL into the original file, it's the recommended way to do it since it's embedded in the original file, you don't need to manage thumbnails subscriptions :

1Thumbnails128Store.onFinishUpload = function(file) {
2    Files.update({_id: file.originalId}, {$set: {thumb128Url: file.url}});
3};
4Thumbnails64Store.onFinishUpload = function(file) {
5    Files.update({_id: file.originalId}, {$set: {thumb64Url: file.url}});
6};

Setting permissions (server)

If you don't want anyone to do anything, you must define permission rules. By default, there is no restriction (except the filter) on insert, remove and update actions.

The permission system has changed since v0.6.1, you must define permissions like this :

1PhotoStore.setPermissions(new UploadFS.StorePermissions({
2    insert(userId, doc) {
3        return userId;
4    },
5    update(userId, doc) {
6        return userId === doc.userId;
7    },
8    remove(userId, doc) {
9        return userId === doc.userId;
10    }
11}));

or when you create the store :

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos',
10    permissions: new UploadFS.StorePermissions({
11        insert(userId, doc) {
12            return userId;
13        },
14        update(userId, doc) {
15            return userId === doc.userId;
16        },
17        remove(userId, doc) {
18            return userId === doc.userId;
19        }
20    })
21});

or you can set default permissions for all stores (since v0.7.1) :

1import {UploadFS} from 'meteor/jalik:ufs';
2
3UploadFS.config.defaultStorePermissions = new UploadFS.StorePermissions({
4    insert(userId, doc) {
5        return userId;
6    },
7    update(userId, doc) {
8        return userId === doc.userId;
9    },
10    remove(userId, doc) {
11        return userId === doc.userId;
12    }
13});

Securing file access (server)

When returning the file for a HTTP request, you can do some checks to decide whether or not the file should be sent to the client. This is done by defining the onRead() method on the store.

Note: Since v0.3.5, every file has a token attribute when its transfer is complete, this token can be used as a password to access/display the file. Just be sure to not publish it if not needed. You can also change this token whenever you want making older links to be staled.

1{{#with image}}
2<a href="{{url}}?token={{token}}">
3    <img src="{{url}}?token={{token}}">
4</a>
5{{/with}}

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos',
10    onRead(fileId, file, request, response) {
11        // Allow file access if not private or if token is correct
12        if (file.isPublic || request.query.token === file.token) {
13            return true;
14        } else {
15            response.writeHead(403);
16            return false;
17        }
18    }
19});

Handling store events and errors (client/server)

Some events are triggered to allow you to do something at the right moment on server side.

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos',
10    // Called when file has been uploaded
11    onFinishUpload(file) {
12        console.log(file.name + ' has been uploaded');
13    },
14    // Called when a copy error happened
15    onCopyError(err, fileId, file) {
16        console.error('Cannot create copy ' + file.name);
17    },
18    // Called when a read error happened
19    onReadError(err, fileId, file) {
20        console.error('Cannot read ' + file.name);
21    },
22    // Called when a write error happened
23    onWriteError(err, fileId, file) {
24        console.error('Cannot write ' + file.name);
25    }
26});

Reading a file from a store (server)

If you need to get a file directly from a store, do like below :

1import {Meteor} from 'meteor/meteor';
2import {Mongo} from 'meteor/mongo';
3import {UploadFS} from 'meteor/jalik:ufs';
4
5const Photos = new Mongo.Collection('photos');
6
7const PhotoStore = new UploadFS.store.Local({
8    collection: Photos,
9    name: 'photos',
10    path: '/uploads/photos'
11});
12
13// Get the file from database
14let file = Photos.findOne({_id: fileId});
15
16// Get the file stream from the store
17let readStream = PhotoStore.getReadStream(fileId, file);
18
19readStream.on('error', Meteor.bindEnvironment(function (error) {
20    console.error(err);
21}));
22readStream.on('data', Meteor.bindEnvironment(function (data) {
23    // handle the data
24}));

Writing a file to a store (server)

If you need to save a file directly to a store, do like below :

1// Insert the file in database
2let fileId = store.create(file);
3
4// Save the file to the store
5store.write(stream, fileId, function(err, file) {
6    if (err) {
7        console.error(err);
8    }else {
9        console.log('file saved to store');
10    }
11});

Uploading files

Uploading from a local file (client)

When the store on the server is configured, you can upload files to it.

Here is the template to upload one or more files :

1<template name="upload">
2    <button type="button" name="upload">Select files</button>
3</template>

And there the code to upload the selected files :

1import {Mongo} from 'meteor/mongo';
2import {Template} from 'meteor/templating';
3import {UploadFS} from 'meteor/jalik:ufs';
4
5const Photos = new Mongo.Collection('photos');
6
7const PhotoStore = new UploadFS.store.Local({
8    collection: Photos,
9    name: 'photos',
10    path: '/uploads/photos'
11});
12
13Template.upload.events({
14    'click button[name=upload]'(ev) {
15        let self = this;
16
17        UploadFS.selectFiles(function (file) {
18            // Prepare the file to insert in database, note that we don't provide a URL,
19            // it will be set automatically by the uploader when file transfer is complete.
20            let photo = {
21                name: file.name,
22                size: file.size,
23                type: file.type,
24                customField1: 1337,
25                customField2: {
26                    a: 1,
27                    b: 2
28                }
29            };
30
31            // Create a new Uploader for this file
32            let uploader = new UploadFS.Uploader({
33                // This is where the uploader will save the file
34                // since v0.6.7, you can pass the store instance or the store name directly
35                store: PhotoStore || 'photos',
36                // Optimize speed transfer by increasing/decreasing chunk size automatically
37                adaptive: true,
38                // Define the upload capacity (if upload speed is 1MB/s, then it will try to maintain upload at 80%, so 800KB/s)
39                // (used only if adaptive = true)
40                capacity: 0.8, // 80%
41                // The size of each chunk sent to the server
42                chunkSize: 8 * 1024, // 8k
43                // The max chunk size (used only if adaptive = true)
44                maxChunkSize: 128 * 1024, // 128k
45                // This tells how many tries to do if an error occurs during upload
46                maxTries: 5,
47                // The File/Blob object containing the data
48                data: file,
49                // The document to save in the collection
50                file: photo,
51                // The error callback
52                onError(err) {
53                    console.error(err);
54                },
55                onAbort(file) {
56                    console.log(file.name + ' upload has been aborted');
57                },
58                onComplete(file) {
59                    console.log(file.name + ' has been uploaded');
60                },
61                onCreate(file) {
62                    console.log(file.name + ' has been created with ID ' + file._id);
63                },
64                onProgress(file, progress) {
65                    console.log(file.name + ' ' + (progress*100) + '% uploaded');
66                },
67                onStart(file) {
68                    console.log(file.name + ' started');
69                },
70                onStop(file) {
71                    console.log(file.name + ' stopped');
72                },
73            });
74
75            // Starts the upload
76            uploader.start();
77
78            // Stops the upload
79            uploader.stop();
80
81            // Abort the upload
82            uploader.abort();
83        });
84    }
85});

Notice : You can use UploadFS.selectFile(callback) or UploadFS.selectFiles(callback) to select one or multiple files, the callback is called with one argument that represents the File/Blob object for each selected file.

During uploading you can get some kind of useful information like the following :

uploader.getAverageSpeed() returns the average speed in bytes per second
uploader.getElapsedTime() returns the elapsed time in milliseconds
uploader.getRemainingTime() returns the remaining time in milliseconds
uploader.getSpeed() returns the speed in bytes per second

Importing file from a URL (server)

You can import a file from an absolute URL by using one of the following methods :

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.store.Local({
7    collection: Photos,
8    name: 'photos',
9    path: '/uploads/photos'
10});
11
12let url = 'https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png';
13let attr = { name: 'Google Logo', description: 'Logo from www.google.com' };
14
15// You can import directly from the store instance
16PhotoStore.importFromURL(url, attr, function (err, file) {
17    if (err) {
18        console.error(err.message);
19    } else {
20        console.log('Photo saved :', file);
21    }
22});
23
24// Or using the common UFS method (requires the store name)
25UploadFS.importFromURL(url, attr, storeName, callback);

WARNING: File type detection is based on the file name in the URL so if the URL is obfuscated it won't work (http://www.hello.com/images/123456 won't work, http://www.hello.com/images/123456.png will work).

NOTE: since v0.6.8, all imported files have the originalUrl attribute, this allows to know where the file comes from and also to do some checks before inserting the file with the StorePermissions.insert.

1import {Mongo} from 'meteor/mongo';
2import {UploadFS} from 'meteor/jalik:ufs';
3
4const Photos = new Mongo.Collection('photos');
5
6const PhotoStore = new UploadFS.Store.Local({
7    name: 'photos',
8    collection: Photos,
9    permissions: new UploadFS.StorePermissions({
10        insert: function(userId, file) {
11            // Check if the file is imported from a URL
12            if (typeof file.originalUrl === 'string') {
13                // allow or not by doing some checks
14            }
15        }
16    })
17});

Setting MIME types (server)

NOTE: only available since v0.6.9

UploadFS automatically detects common MIME types based on the file extension. So when uploading a file, if the file.type is not set, UploadFS will check in its MIME list and assign the corresponding MIME, you can also add your own MIME types.

1import {UploadFS} from 'meteor/jalik:ufs';
2
3// Adds KML and KMZ MIME types detection
4UploadFS.addMimeType('kml', 'application/vnd.google-earth.kml+xml');
5UploadFS.addMimeType('kmz', 'application/vnd.google-earth.kmz');

If you want to get all MIME types :

1import {UploadFS} from 'meteor/jalik:ufs';
2
3const MIME = UploadFS.getMimeTypes();

Displaying images (client)

To display a file, simply use the url attribute for an absolute URL or the path attribute for a relative URL.

NOTE: path is only available since v0.6.8

You can upgrade existing documents to add the path attribute by calling the UploadFS.addPathAttributeToFiles(where) which will upgrade all collections linked to any UploadFS store, the where option is not required, default predicate is {path: null}.

Here is the template to display a list of photos :

1<template name="photos">
2    <div>
3        {{#each photos}}
4            {{#if uploading}}
5                <img src="/images/spinner.gif" title="{{name}}">
6                <span>{{completed}}%</span>
7            {{else}}
8                <img src="{{url}}" title="{{name}}">
9            {{/if}}
10        {{/each}}
11    </div>
12</template>

And there the code to load the file :

1Template.photos.helpers({
2    completed: function() {
3        return Math.round(this.progress * 100);
4    },
5    photos: function() {
6        return Photos.find();
7    }
8});

Using template helpers (client)

Some helpers are available by default to help you work with files inside templates.

1{{#if isApplication}}
2    <a href="{{url}}">Download</a>
3{{/if}}
4{{#if isAudio}}
5    <audio src="{{url}}" controls></audio>
6{{/if}}
7{{#if isImage}}
8    <img src="{{url}}">
9{{/if}}
10{{#if isText}}
11    <iframe src={{url}}></iframe>
12{{/if}}
13{{#if isVideo}}
14    <video src="{{url}}" controls></video>
15{{/if}}

Changelog

Version 0.7.2

Adds attribute etag to uploaded files
Adds HTTP cache support : return HTTP code 304 depending of Last-Modified and If-None-Match request headers (#110)
Adds option Store.onValidate(file) to validate a file before writing to the store
Adds method UploadFS.addETagAttributeToFiles(where) to add etag attribute to existing files
Adds method UploadFS.generateEtag()
Adds method Store.validate(file)
Uses ES6 class syntax
Uses ES6 import syntax

Version 0.7.1

Adds default store permissions in UploadFS.config.defaultStorePermissions
Fixes store permissions (#95)
Fixes HTTP Range result from stream (#94) : works with ufs-local and ufs-gridfs

Version 0.7.0_2

Adds support for Range request headers (to seek audio/video files)
Upgrades dependencies

Version 0.6.9

Adds ufs-mime.js file to handle mime related operations
Sets default file type to "application/octet-stream"
Detects automatically MIME type by checking file extension on upload (#84)
Fixes error thrown by UploadFS.Filter.checkContentType() when file type is empty
Fixes check(file, Object); into "ufsImportURL" method

Version 0.6.8

Passes full predicate in CRUD operations instead of just the ID
Removes file tokens when file is uploaded or removed
Adds the "originalUrl" attribute to files imported from URLs
Adds the "path" attribute to uploaded files corresponding to the relative URL of the file
Adds UploadFS.Store.prototype.getRelativeURL(path) to get the relative URL of a store
Adds UploadFS.Store.prototype.getFileRelativeURL(path) to get the relative URL of a file in a store
Adds UploadFS.addPathAttributeToFiles(where) to add the path attribute to existing files
Unblock the UploadFS.importFromURL()
Fixes file deletion during upload (stop uploading and removes temp file)

Version 0.6.7

Allows to define stores on server only, use the store name directly as a reference on the client
Fixes an error caused by the use of an upsert in multi-server environment (mongo)

Version 0.6.5

Fixes 504 Gateway timeout error when file is not served by UploadFS

Version 0.6.4

Allows to set temp directory permissions

Version 0.6.3

Fixes iOS and Android issues
Adds CORS support

Version 0.6.1

Brings a huge improvement to large file transfer
Uses POST HTTP method instead of Meteor methods to upload files
Simplifies code for future versions

Breaking changes

UploadFS.readAsArrayBuffer() is DEPRECATED

The method UploadFS.readAsArrayBuffer() is not available anymore, as uploads are using POST binary data, we don't need ArrayBuffer.

1import {UploadFS} from 'meteor/jalik:ufs';
2
3UploadFS.selectFiles(function(ev){
4    UploadFS.readAsArrayBuffer(ev, function (data, file) {
5        let photo = {
6            name: file.name,
7            size: file.size,
8            type: file.type
9        };
10        let worker = new UploadFS.Uploader({
11            store: PhotoStore,
12            data: data,
13            file: photo
14        });
15        worker.start();
16    });
17});

The new code is smaller and easier to read :

1import {UploadFS} from 'meteor/jalik:ufs';
2
3UploadFS.selectFiles(function(file){
4    let photo = {
5        name: file.name,
6        size: file.size,
7        type: file.type
8    };
9    let worker = new UploadFS.Uploader({
10        store: PhotoStore,
11        data: file,
12        file: photo
13    });
14    worker.start();
15});

Permissions are defined differently

Before v0.6.1 you would do like this :

1Photos.allow({
2    insert(userId, doc) {
3        return userId;
4    },
5    update(userId, doc) {
6        return userId === doc.userId;
7    },
8    remove(userId, doc) {
9        return userId === doc.userId;
10    }
11});

Now you can set the permissions when you create the store :

1PhotoStore = new UploadFS.store.Local({
2    collection: Photos,
3    name: 'photos',
4    path: '/uploads/photos',
5    permissions: new UploadFS.StorePermissions({
6        insert(userId, doc) {
7            return userId;
8        },
9        update(userId, doc) {
10            return userId === doc.userId;
11        },
12        remove(userId, doc) {
13            return userId === doc.userId;
14        }
15    })
16});

License

UploadFS is released under the MIT License.