aldeed:collection2-core

A Meteor package that allows you to attach a schema to a Mongo.Collection. Automatically validates against that schema when inserting and updating from client or server code.

This package requires the simpl-schema NPM package, which defines the schema syntax and provides the validation logic.

Installation

In your Meteor app directory, enter:

$ meteor add aldeed:collection2-core
$ meteor npm install --save simpl-schema

Why Use Collection2?

• While adding allow/deny rules ensures that only authorized users can edit a

document from the client, adding a schema ensures that only acceptable properties and values can be set within that document from the client. Thus, client side inserts and updates can be allowed without compromising security or data integrity.

• Schema validation for all inserts and updates is reactive, allowing you to

easily display customizable validation error messages to the user without any event handling.

• Schema validation for all inserts and updates is automatic on both the client

and the server, providing both speed and security.

• The aldeed:autoform package can

take your collection's schema and automatically create HTML5 forms based on it. AutoForm provides automatic database operations, method calls, validation, and user interface reactivity. You have to write very little markup and no event handling. Refer to the AutoForm documentation for more information.

Attaching a Schema to a Collection

Let's say we have a normal "books" collection, defined in code that runs on both client and server (common.js):

1const Books = new Mongo.Collection("books");

Let's create a SimpleSchema schema for this collection. We'll do this in common.js, too:

1const Schemas = {};
2
3Schemas.Book = new SimpleSchema({
4    title: {
5        type: String,
6        label: "Title",
7        max: 200
8    },
9    author: {
10        type: String,
11        label: "Author"
12    },
13    copies: {
14        type: SimpleSchema.Integer,
15        label: "Number of copies",
16        min: 0
17    },
18    lastCheckedOut: {
19        type: Date,
20        label: "Last date this book was checked out",
21        optional: true
22    },
23    summary: {
24        type: String,
25        label: "Brief summary",
26        optional: true,
27        max: 1000
28    }
29});

Once we have the SimpleSchema instance, all we need to do is attach it to our collection using the attachSchema method. Again, we will do this in common.js:

1Books.attachSchema(Schemas.Book);

Now that our collection has a schema, we can do a validated insert on either the client or the server:

1Books.insert({title: "Ulysses", author: "James Joyce"}, (error, result) => {
2  //The insert will fail, error will be set,
3  //and result will be undefined or false because "copies" is required.
4  //
5  //The list of errors is available on `error.invalidKeys` or by calling Books.simpleSchema().namedContext().validationErrors()
6});

Or we can do a validated update:

1Books.update(book._id, {$unset: {copies: 1}}, (error, result) => {
2  //The update will fail, error will be set,
3  //and result will be undefined or false because "copies" is required.
4  //
5  //The list of errors is available on `error.invalidKeys` or by calling Books.simpleSchema().namedContext().validationErrors()
6});

Attaching Multiple Schemas to the Same Collection

Normally, if call attachSchema multiple times, the schemas are merged. If you use the replace: true option, then it will replace the previously attached schema. However, in some cases you might actually want both schemas attached, with different documents validated against different schemas.

Here is an example:

1Products.attachSchema(SimpleProductSchema, {selector: {type: 'simple'}});
2Products.attachSchema(VariantProductSchema, {selector: {type: 'variant'}});

Now both schemas are attached. When you insert a document where type: 'simple' in the document, it will validate against only the SimpleProductSchema. When you insert a document where type: 'variant' in the document, it will validate against only the VariantProductSchema.

Alternatively, you can pass a selector option when inserting to choose which schema to use:

1Products.insert({ title: 'This is a product' }, { selector: { type: 'simple' } });

For an update or upsert, the matching selector can be in the query, the modifier $set object, or the selector option.

attachSchema options

transform

If your validation requires that your doc be transformed using the collection's transform function prior to being validated, then you must pass the transform: true option to attachSchema when you attach the schema:

1Books.attachSchema(Schemas.Book, {transform: true});

replace

By default, if a collection already has a schema attached, attachSchema will combine the new schema with the existing. Pass the replace: true option to attachSchema to discard any existing schema.

Attach a Schema to Meteor.users

Obviously, when you attach a schema, you must know what the schema should be. For Meteor.users, here is an example schema, which you might have to adjust for your own needs:

1const Schema = {};
2
3Schema.UserCountry = new SimpleSchema({
4    name: {
5        type: String
6    },
7    code: {
8        type: String,
9        regEx: /^[A-Z]{2}$/
10    }
11});
12
13Schema.UserProfile = new SimpleSchema({
14    firstName: {
15        type: String,
16        optional: true
17    },
18    lastName: {
19        type: String,
20        optional: true
21    },
22    birthday: {
23        type: Date,
24        optional: true
25    },
26    gender: {
27        type: String,
28        allowedValues: ['Male', 'Female'],
29        optional: true
30    },
31    organization : {
32        type: String,
33        optional: true
34    },
35    website: {
36        type: String,
37        regEx: SimpleSchema.RegEx.Url,
38        optional: true
39    },
40    bio: {
41        type: String,
42        optional: true
43    },
44    country: {
45        type: Schema.UserCountry,
46        optional: true
47    }
48});
49
50Schema.User = new SimpleSchema({
51    username: {
52        type: String,
53        // For accounts-password, either emails or username is required, but not both. It is OK to make this
54        // optional here because the accounts-password package does its own validation.
55        // Third-party login packages may not require either. Adjust this schema as necessary for your usage.
56        optional: true
57    },
58    emails: {
59        type: Array,
60        // For accounts-password, either emails or username is required, but not both. It is OK to make this
61        // optional here because the accounts-password package does its own validation.
62        // Third-party login packages may not require either. Adjust this schema as necessary for your usage.
63        optional: true
64    },
65    "emails.$": {
66        type: Object
67    },
68    "emails.$.address": {
69        type: String,
70        regEx: SimpleSchema.RegEx.Email
71    },
72    "emails.$.verified": {
73        type: Boolean
74    },
75    // Use this registered_emails field if you are using splendido:meteor-accounts-emails-field / splendido:meteor-accounts-meld
76    registered_emails: {
77        type: Array,
78        optional: true
79    },
80    'registered_emails.$': {
81        type: Object,
82        blackbox: true
83    },
84    createdAt: {
85        type: Date
86    },
87    profile: {
88        type: Schema.UserProfile,
89        optional: true
90    },
91    // Make sure this services field is in your schema if you're using any of the accounts packages
92    services: {
93        type: Object,
94        optional: true,
95        blackbox: true
96    },
97    // Add `roles` to your schema if you use the meteor-roles package.
98    // Option 1: Object type
99    // If you specify that type as Object, you must also specify the
100    // `Roles.GLOBAL_GROUP` group whenever you add a user to a role.
101    // Example:
102    // Roles.addUsersToRoles(userId, ["admin"], Roles.GLOBAL_GROUP);
103    // You can't mix and match adding with and without a group since
104    // you will fail validation in some cases.
105    roles: {
106        type: Object,
107        optional: true,
108        blackbox: true
109    },
110    // Option 2: [String] type
111    // If you are sure you will never need to use role groups, then
112    // you can specify [String] as the type
113    roles: {
114        type: Array,
115        optional: true
116    },
117    'roles.$': {
118        type: String
119    },
120    // In order to avoid an 'Exception in setInterval callback' from Meteor
121    heartbeat: {
122        type: Date,
123        optional: true
124    }
125});
126
127Meteor.users.attachSchema(Schema.User);

This schema has not been thoroughly vetted to ensure that it accounts for all possible properties the accounts packages might try to set. Furthermore, any other packages you add might also try to set additional properties. If you see warnings in the console about keys being removed, that's a good indication that you should add those keys to the schema.

Note also that this schema uses the blackbox: true option for simplicity. You might choose instead to figure out a more specific schema.

(If you figure out a more accurate Meteor.users schema, documentation pull requests are welcome.)

Schema Format

Refer to the simpl-schema package documentation for a list of all the available schema rules and validation methods.

Use the MyCollection.simpleSchema() method to access the attached SimpleSchema instance for a Mongo.Collection instance. For example:

1MyCollection.simpleSchema().validate(doc);

Passing Options

In Meteor, the update function accepts an options argument. Collection2 changes the insert function signature to also accept options in the same way, as an optional second argument. Whenever this documentation says to "use X option", it's referring to this options argument. For example:

1myCollection.insert(doc, {validate: false});

Validation Contexts

In the examples above, note that we called namedContext() with no arguments to access the SimpleSchema reactive validation methods. Contexts let you keep multiple separate lists of invalid keys for a single collection. In practice you might be able to get away with always using the default context. It depends on what you're doing. If you're using the context's reactive methods to update UI elements, you might find the need to use multiple contexts. For example, you might want one context for inserts and one for updates, or you might want a different context for each form on a page.

To use a specific named validation context, use the validationContext option when calling insert or update:

1Books.insert({title: "Ulysses", author: "James Joyce"}, { validationContext: "insertForm" }, (error, result) => {
2  //The list of errors is available by calling Books.simpleSchema().namedContext("insertForm").validationErrors()
3});
4
5Books.update(book._id, {$unset: {copies: 1}}, { validationContext: "updateForm" }, (error, result) => {
6  //The list of errors is available by calling Books.simpleSchema().namedContext("updateForm").validationErrors()
7});

Validating Without Inserting or Updating

It's also possible to validate a document without performing the actual insert or update:

1Books.simpleSchema().namedContext().validate({title: "Ulysses", author: "James Joyce"}, {modifier: false});

Set the modifier option to true if the document is a mongo modifier object.

You can also validate just one key in the document:

1Books.simpleSchema().namedContext().validate({title: "Ulysses", author: "James Joyce"}, {modifier: false, keys: ['title']});

Or you can specify a certain validation context when calling either method:

1Books.simpleSchema().namedContext("insertForm").validate({title: "Ulysses", author: "James Joyce"}, {modifier: false});
2Books.simpleSchema().namedContext("insertForm").validate({title: "Ulysses", author: "James Joyce"}, {modifier: false, keys: ['title']});

Refer to the simpl-schema package documentation for more information about these methods.

Inserting or Updating Without Validating

To skip validation, use the validate: false option when calling insert or update. On the client (untrusted code), this will skip only client-side validation. On the server (trusted code), it will skip all validation. The object is still cleaned and autoValues are still generated.

Inserting or Updating Without Cleaning

Skip removing properties that are not in the schema

To skip object property filtering, set the filter option to false when you call insert or update.

Skip conversion of values to match what schema expects

To skip automatic value conversion, set the autoConvert option to false when you call insert or update.

Skip removing empty strings

To skip removing empty strings, set the removeEmptyStrings option to false when you call insert or update.

Skip generating automatic values

To skip adding automatic values, set the getAutoValues option to false when you call insert or update. This works only in server code.

Inserting or Updating Bypassing Collection2 Entirely

Even if you skip all validation and cleaning, Collection2 will still do some object parsing that can take a long time for a large document. To bypass this, set the bypassCollection2 option to true when you call insert or update. This works only in server code.

Additional SimpleSchema Options

In addition to all the other schema validation options documented in the simpl-schema package, the collection2 package adds additional options explained in this section.

index and unique

See https://github.com/aldeed/meteor-schema-index

denyInsert and denyUpdate

See https://github.com/aldeed/meteor-schema-deny

autoValue

The autoValue option is provided by the SimpleSchema package and is documented there. Collection2 adds the following properties to this for any autoValue function that is called as part of a C2 database operation:

• isInsert: True if it's an insert operation
• isUpdate: True if it's an update operation
• isUpsert: True if it's an upsert operation (either upsert() or upsert: true)
• userId: The ID of the currently logged in user. (Always null for server-initiated actions.)
• isFromTrustedCode: True if the insert, update, or upsert was initiated from trusted (server) code
• docId: The _id property of the document being inserted or updated. For an insert, this will be set only when it is provided in the insert doc, or when the operation is initiated on the client. For an update or upsert, this will be set only when the selector is or includes the _id, or when the operation is initiated on the client.

Note that autoValue functions are run on the client only for validation purposes, but the actual value saved will always be generated on the server, regardless of whether the insert/update is initiated from the client or from the server.

There are many possible use cases for autoValue. It's probably easiest to explain by way of several examples:

1{
2  // Force value to be current date (on server) upon insert
3  // and prevent updates thereafter.
4  createdAt: {
5    type: Date,
6    autoValue: function() {
7      if (this.isInsert) {
8        return new Date();
9      } else if (this.isUpsert) {
10        return {$setOnInsert: new Date()};
11      } else {
12        this.unset();  // Prevent user from supplying their own value
13      }
14    }
15  },
16  // Force value to be current date (on server) upon update
17  // and don't allow it to be set upon insert.
18  updatedAt: {
19    type: Date,
20    autoValue: function() {
21      if (this.isUpdate) {
22        return new Date();
23      }
24    },
25    denyInsert: true,
26    optional: true
27  },
28  // Whenever the "content" field is updated, automatically store
29  // the first word of the content into the "firstWord" field.
30  firstWord: {
31    type: String,
32    optional: true,
33    autoValue: function() {
34      var content = this.field("content");
35      if (content.isSet) {
36        return content.value.split(' ')[0];
37      } else {
38        this.unset();
39      }
40    }
41  },
42  // Whenever the "content" field is updated, automatically
43  // update a history array.
44  updatesHistory: {
45    type: Array,
46    optional: true,
47    autoValue: function() {
48      var content = this.field("content");
49      if (content.isSet) {
50        if (this.isInsert) {
51          return [{
52              date: new Date(),
53              content: content.value
54            }];
55        } else {
56          return {
57            $push: {
58              date: new Date,
59              content: content.value
60            }
61          };
62        }
63      } else {
64        this.unset();
65      }
66    }
67  },
68  'updatesHistory.$': {
69    type: Object,
70  },
71  'updatesHistory.$.date': {
72    type: Date,
73    optional: true
74  },
75  'updatesHistory.$.content': {
76    type: String,
77    optional: true
78  },
79  // Automatically set HTML content based on markdown content
80  // whenever the markdown content is set.
81  htmlContent: {
82    type: String,
83    optional: true,
84    autoValue: function(doc) {
85      var markdownContent = this.field("markdownContent");
86      if (Meteor.isServer && markdownContent.isSet) {
87        return MarkdownToHTML(markdownContent.value);
88      }
89    }
90  }
91}

custom

The custom option is provided by the SimpleSchema package and is documented there. Collection2 adds the following properties to this for any custom function that is called as part of a C2 database operation:

• isInsert: True if it's an insert operation
• isUpdate: True if it's an update operation
• isUpsert: True if it's an upsert operation (either upsert() or upsert: true)
• userId: The ID of the currently logged in user. (Always null for server-initiated actions.)
• isFromTrustedCode: True if the insert, update, or upsert was initiated from trusted (server) code
• docId: The _id property of the document being inserted or updated. For an insert, this will be set only when it is provided in the insert doc, or when the operation is initiated on the client. For an update or upsert, this will be set only when the selector is or includes the _id, or when the operation is initiated on the client.

What Happens When The Document Is Invalid?

The callback you specify as the last argument of your insert() or update() call will have the first argument (error) set to an Error instance. The error message for the first invalid key is set in the error.message, and the full validationErrors array is available on error.invalidKeys. This is true on both client and server, even if validation for a client-initiated operation does not fail until checked on the server.

If you attempt a synchronous operation in server code, the same validation error is thrown since there is no callback to pass it to. If this happens in a server method (defined with Meteor.methods), a more generic Meteor.Error is passed to your callback back on the client. This error does not have an invalidKeys property, but it does have the error message for the first invalid key set in error.reason.

Generally speaking, you would probably not use the Error for displaying to the user. You can instead use the reactive methods provided by the SimpleSchema validation context to display the specific error messages to the user somewhere in the UI. The autoform package provides some UI components and helpers for this purpose.

More Details

For the curious, this is exactly what Collection2 does before every insert or update:

1. Removes properties from your document or mongo modifier object if they are

not explicitly listed in the schema. (To skip this, set the filter option to false when you call insert or update.) 2. Automatically converts some properties to match what the schema expects, if possible. (To skip this, set the autoConvert option to false when you call insert or update.) 3. Optimizes your operation so that empty string values will not be stored. (To skip this, set the removeEmptyStrings option to false when you call insert or update.) 3. Adds automatic (forced or default) values based on your schema. Values are added only on the server and will make their way back to your client when your subscription is updated. (To skip this in server code, set the getAutoValues option to false when you call insert or update.) 4. Validates your document or mongo modifier object. (To skip this, set the validate option to false when you call insert or update.) 5. Performs the insert or update like normal, only if it was valid.

Collection2 is simply calling SimpleSchema methods to do these things. The validation happens on both the client and the server for client-initiated actions, giving you the speed of client-side validation along with the security of server-side validation.

Community Add-On Packages

Automatic Migrations

The davidyaha:collection2-migrations package can watch for schema changes between server restarts and perform some automatic data migration and cleanup.

Problems?

You might find yourself in a situation where it seems as though validation is not working correctly. First, you should enable SimpleSchema debug mode by setting SimpleSchema.debug = true, which may log some additional information. If you're still confused, read through the following tricky, confusing situations.

SubObjects and Arrays of Objects

One critical thing to know about Collection2 and SimpleSchema is that they don't validate the saved document but rather the proposed insert doc or the update modifier. In the case of updates, this means there is some information unknown to SimpleSchema, such as whether the array object you're attempting to modify already exists or not. If it doesn't exist, MongoDB would create it, so SimpleSchema will validate conservatively. It will assume that any properties not set by the modifier will not exist after the update. This means that the modifier will be deemed invalid if any required keys in the same object are not explicitly set in the update modifier.

For example, say we add the following keys to our "books" schema:

1{
2    borrowedBy: {
3        type: Array
4    },
5    'borrowedBy.$': {
6        type: Object
7    },
8    "borrowedBy.$.name": {
9        type: String
10    },
11    "borrowedBy.$.email": {
12        type: String,
13        regEx: SimpleSchema.RegEx.Email
14    },
15}

Every object in the borrowedBy array must have a name and email property.

Now we discover that the name is incorrect in item 1, although the email address is correct. So we will just set the name to the correct value:

1Books.update(id, {$set: {"borrowedBy.1.name": "Frank"}});

However, this will not pass validation. Why? Because we don't know whether item 1 in the borrowedBy array already exists, so we don't know whether it will have the required email property after the update finishes.

There are three ways to make this work:

• $set the entire object
• $set all required keys in the object
• Perform the update on the server, and pass the validate: false option to skip validation.

When this situation occurs on the client with an autoForm, it generally does not cause any problems because AutoForm is smart enough to $set the entire object; it's aware of this potential issue. However, this means that you need to ensure that all required properties are represented by an input on the form. In our example, if you want an autoForm that only shows a field for changing the borrowedBy name and not the email, you should include both fields but make the email field hidden. Alternatively, you can submit the autoForm to a server method and then do a server update without validation.

Although these examples focused on an array of objects, sub-objects are treated basically the same way.

Contributing

Anyone is welcome to contribute. Fork, make and test your changes (meteor test-packages ./), and then submit a pull request.

Running Tests

$ cd tests
$ meteor npm i && npm test

Running Tests in Watch Mode

$ cd tests
$ meteor npm i && npm run test:watch

Major Contributors

@mquandalle

(Add yourself if you should be listed here.)