Meteor Grid-Factory

GitHub

Create FilesCollections with integrated GridFS storage. Lightweight. Simple.

With this package you can easily create multiple ostrio:files collections (FilesCollections) that work with MongoDB's GridFS system out-of-the-box.

Background / reasons

It can be a real hassle to introduce gridFS as storage to your project. This package aims to abstract common logic into an easy and accessible API while ensuring to let you override anything in case you need a fine-tuned custom behavior.

The abtract factory allows you to create configurations on a higher level that apply to all your FilesCollections, while you still can fine-tune on the collection level. Supports all constructor arguments of FilesCollection.

Why such a specialized package?
- What is covered / what is not (yet)
Getting startedvalidateUser
Changelog
License

Why such a specialized package?

This package is designed for projects, that can't rely on third-party storages, because one or more of the following applies:

there are concerns about privacy or security when using a third party storage
the application has to be shipped in a "all-in-one" monolith
the app is intended for intranet use and connection to the "outside" is prohibited
whatever you can think of....

The use case may be not very common (Meteor + ostrio:files + GridFS) but if it's for you, this package makes file handling much easier and consistent.

What is covered by this package

This package has some out-of-the-box functionality that covers the following points.

Creation

[ x ] Creating new FilesCollection instances
[ x ] Supporting full FilesCollection constructor
[ x ] Allows to override package internals by explicitly passing the hooks (onAfterUpload etc.)
[ x ] Code-splitting (server/client)
[ x ] Using GridFS buckets instead of deprecated gridfs-stream
[ x ] Adapter for translation
[ x ] Creating new FilesCollection instances

`onBeforeUpload`

This package has some some default behavior defined for the onBeforeUpload hook. You can override it completely or hook into it's behavior using the following parameters:

[ x ] check file size via maxSize (Number)
[ x ] check file extensions via extensions ([String])
[ x ] check permissions via validateUser (Function)
[ x ] check permissions via validateUser (Function)

`onAfterUpload`

The default behavior for onAfterUpload is to check the mime of the uploaded file and move it to the Grid. However, you can hook into this process, too:

[ x ] validate user via validateUser (Function)
[ x ] validate mime via validateMime (Function)
[ x ] transform additional versions (e.g. thumbnails, converted videos, etc.) via transformVersions (Function)

`protected`

[ x ] validate user via validateUser (Function)

`interceptDownload`

[ x ] falls back to find a valid version, if request to a non-existent version fails
[ x ] streams the file from the GridFS bucket
[ x ] handles errors with an error response
[ x ] sets the correct content disposition, depending on download query attribute
[ x ] 206 response streaming

`onBeforeRemove`

[ x ] validate user via validateUser (Function)

`afterRemove`

[ x ] removes file, including all versions, from the GridFS and the FilesCollection

Getting started

1. Install this package via

meteor add leaonline:files-collection-factory ostrio:files

We decoupled this package from ostrio:files so your host project can manage the versioning.

2. Optionally install packages for mime-check and transformations

If you want to check the mime you can use packages like mmmagic and mime-types to check for the correct mime type. Of course you can implement your mime-check a total different way, too.

meteor npm install --save mmmagic mime-types

If you want to transform your images or videos you also need the respective packages for that. Often you will also have to install additional software / packages on your host OS, since the npm packages (e.g. for image magick / graphics magic) are usually just wrappers for the OS-level packages.

3. Import the abstract factory

The package exports different APIs for client and server but you import it the same way on server and client:

1import { createGridFilesFactory } from 'meteor/leaonline:grid-factory'

From here you have to consider the FilesCollection architecture in order to manage access, post processing, removal etc.

4. Create a server side FilesCollection factory

On the server side you can use the following abstract factory api:

1({
2  i18nFactory: Function, // translator function, str => translatedStr
3  fs: Object, // the node file-system module
4  bucketFactory: Function, // a function that returns a gridFS bucket
5  defaultBucket: String, // a default name for the bucket to be used
6  createObjectId: Function, // a function that creates an Object Id by a given GridFS id
7  onError: Function, // logs errors for all collections across all factories
8  debug: Boolean,
9    ...config // all valid config, that can be passed to the FilesCollection server constructor
10}) => Function => FilesCollection

The factory Function that is returned contains the following api:

1({
2  bucketName: String, // override the defaultBucket, if desired
3  maxSize: Number, // number in bytes to limit the maximum size for files of this collection
4  extensions: [String], // a list of supported extensions
5  validateUser: Function, // a Function that checks permission of the current user/file and returns falsy/truthy
6  validateMime: Function, // async Function that checks permission of the current file/mime and returns falsy/truthy
7  transformVersions: Function, // async Function that transforms the file to different versions
8  onError: Function // logs errors, overrides onError from abstract factory
9}) => FilesCollection

Minimal example

The following example shows a minimal abstract GridFactory:

1import { MongoInternals } from 'meteor/mongo'
2import { createGridFilesFactory } from 'meteor/leaonline:grid-factory'
3import { i18n } from '/path/to/i8n'
4import fs from 'fs'
5
6const debug = Meteor.isDevelopment
7const i18nFactory = (...args) => i18n.get(...args)
8const createObjectId = ({ gridFsFileId }) => new MongoInternals.NpmModule.ObjectID(gridFsFileId)
9const bucketFactory = bucketName => 
10  new MongoInternals.NpmModule.GridFSBucket(MongoInternals.defaultRemoteCollectionDriver().mongo.db, { bucketName })
11const defaultBucket = 'fs' // resolves to fs.files / fs.chunks as default
12const onError = error => console.error(error)
13
14const createFilesCollection = createGridFilesFactory({ 
15  i18nFactory, 
16  fs, 
17  bucketFactory, 
18  defaultBucket, 
19  createObjectId, 
20  onError, 
21  debug 
22})
23
24const ProfileImages = createFilesCollection({
25  collectionName: 'profileImages',
26  bucketName: 'images', // put image collections in the 'images' bucket
27  maxSize: 3072000, // 3 MB max
28  validateUser: function (userId, file, type, translate) {
29    // is this a valid and registered user?
30    if (!userId || Meteor.users.find(userId).count() !== 1) {
31      return false
32    }
33    
34    const isOwner = userId === file.userId
35    const isAdmin = ...
36    const isAllowedToDownload =  ...
37    
38    if (type === 'upload') {
39      return Roles.userIsInRole(userId, 'can-upload', 'mydomain.com') // example of using roles
40    }
41    
42    if (type === 'download') {
43      return isOwner || isAdmin || isAllowedToDownload // custom flags
44    }
45    
46    if (type === 'remove') {
47     // allow only owner to remove the file
48     return isOwner || isAdmin
49    }
50    
51    throw new Error(translate('unexpectedCodeReach'))
52  }
53})

5. Create a client side factory

On the server side you have less options to pass to the API:

1({
2  i18nFactory: Function, // translator function, str => translatedStr
3  debug: Boolean,
4  ...config // all valid config, that can be passed to the FilesCollection client constructor
5}) => Function => FilesCollection

The factory Function that is returned contains the following api:

1({
2  bucketName: String, // override the defaultBucket, if desired
3  maxSize: Number, // number in bytes to limit the maximum size for files of this collection
4  extensions: [String], // a list of supported extensions
5  validateUser: Function, // a Function that checks permission of the current user/file and returns falsy/truthy
6  validateMime: Function, // async Function that checks permission of the current file/mime and returns falsy/truthy
7  transformVersions: Function, // async Function that transforms the file to different versions
8  onError: Function // logs errors, overrides onError from abstract factory
9}) => FilesCollection

Contribution

Contributions are very welcomed! Please leave an issue before creating a PR in order to discuss feasibility and boundaries of a potential PR.

Testing

We provide a special Meteor project for tests for you. I contains scripts for linting and testing out of the box:

cd test-proxy
meteor npm run setup
meteor npm run lint:code
meteor npm run lint:markdown
meteor npm run test:watch

Changelog

1.3.0
- partial response (206) implemented
- onAfterUpload is now fully async with correct order; return value will not be returned until all hooks are called (validateMime, transformVersions, moveToGrid) and completed;
- file has a processingComplete flag that will indicate when the onAfterUpload pipeline has been completed, which allows for example to not link the file on the client until fully processed; make sure your schema supports it!
- tests updated a lot
1.2.0
- security patch to prevent server process crashing if mime check fails
1.1.0
- major bump for ostrio:files to support 1.x AND 2.x
- logging improved
- improved checks and loggins for all library functions
- tests added
- documentation improved
1.0.2
- getGridFsFileId fix bug searching fallback versions
1.0.1
- allow skipping user validation for prototyping but raise a server warning
- README fix export name
- standardjs lint fix

License

MIT, see LICENSE file