publish-relations

Maintained fork of cottz:publish-relations by lfades (MIT). Includes bug fixes and improvements over the original — see the releases.

Edit your documents before sending without too much stress. provides a number of methods to easily manipulate data using internally observe and observeChanges in the server

Installation

$ meteor add danmacko:publish-relations

Publications are then declared with Meteor.publishRelations in place of Meteor.publish. The same function is also the package's default export, so these two are interchangeable:

1Meteor.publishRelations('books', function () { /* ... */ });
2
3import PublishRelations from 'meteor/danmacko:publish-relations';
4PublishRelations('books', function () { /* ... */ });

The examples below use the first: it needs no import beyond the Meteor a publications file has anyway, and it reads next to the Meteor.publish calls it usually sits among. The package also puts a bare PublishRelations global in scope (api.export), which works too but trips eslint's no-undef.

Compatibility


Minimum	Meteor 3.0 — enforced by `api.versionsFrom` in `package.js`
Tested on	Meteor 3.4.1 (both test layers, see Testing)
Meteor 2	the 3.x line, feature-frozen, with the older API — see Upgrading

The install command above is the right one on both lines: api.versionsFrom('3.0') puts 4.x out of reach of a Meteor 2 app's constraint solver, so it resolves the newest 3.x there and the newest 4.x on Meteor 3. Nothing has to be pinned by hand, and an app that already carries a @3.x constraint is free to meteor update into anything else that lands on that line — a Meteor constraint is a floor within the major version, not an exact version (that is @=3.2.1).

Quick Start

Assuming we have the following collections

1// Authors
2{
3  _id: 'someAuthorId',
4  name: 'Luis',
5  profile: 'someProfileId',
6  bio: 'I am a very good and happy author',
7  interests: ['writing', 'reading', 'others']
8}
9
10// Books
11{
12  _id: 'someBookId',
13  authorId: 'someAuthorId',
14  name: 'meteor for dummies'
15}
16
17// Comments
18{
19  _id: 'someCommentId',
20  bookId: 'someBookId',
21  text: 'This book is better than meteor for pros :O'
22}

I want publish the autor with his books and comments of the books

1Meteor.publishRelations('author', function (authorId) {
2  this.relations.cursor(Authors.find(authorId), function (id, doc) {
3    this.relations.cursor(Books.find({authorId: id}), function (id, doc) {
4      this.relations.cursor(Comments.find({bookId: id}));
5    });
6  });
7
8  return this.ready();
9});

That is the shape the problem has, and it works — but it opens a comments query per book, so it gets slower with every book the author writes. Collecting the ids and asking once does the same job:

1Meteor.publishRelations('author', function (authorId) {
2  const comments = this.relations.join(Comments);
3  comments.selector = _ids => ({bookId: _ids});
4
5  this.relations.cursor(Authors.find(authorId), function (id, doc) {
6    this.relations.cursor(Books.find({authorId: id}), function (id, doc) {
7      comments.push(id);
8    });
9  });
10
11  comments.send();
12
13  return this.ready();
14});

Three queries for Mongo however many books there are, and no less reactive: a push that changes what the join holds restarts its single query, coalesced so a burst of them costs one restart. Performance Notes works the arithmetic through and covers the rest of the rules — it is the section to read before writing a publication of your own.

Asynchrony

On Meteor 3 registering an observer is asynchronous — MongoConnection._observeChanges is async, so cursor.observeChanges() hands back a promise rather than a handle. Everything in this package that registers one therefore returns a promise too: relations.cursor(), relations.cursorNonreactive(), relations.observe(), relations.observeChanges() and join.send(). join.push() stays synchronous; it is bookkeeping, not I/O.

You do not have to await any of them. The package waits instead:

this.ready() in a publication body is recorded, not sent. The real ready goes out once every registration the body started has settled — so the client is told "that is everything" when it is, not while observers are still being built. A body that means the literal Meteor behaviour (ready first, data when it comes) calls this.relations.readyNow().
Inside a callback the same holds one level down: what the callback registers is settled before the parent document's own write goes out, so a nested cursor's documents cannot arrive after the document they hang off.
A join first materialises behind the registrations that precede it in the same body, so send() sees the complete membership its callbacks pushed.

This is the ordering Fibers gave for free on Meteor 2. Awaiting is still allowed — the promises are real — but a body written the Meteor 2 way behaves correctly as written, which is what makes upgrading a publication a rename and nothing more.

Callbacks may be async

They often have to be: findOne, insert, update and remove throw on the Meteor 3 server, so any callback that reaches into another collection is async.

This is also the shape to reach for when all a document needs is one field from somewhere else, since a nested cursor would publish the whole joined document to get it:

1this.relations.cursor(Books.find(), async function (id, doc) {
2  // guarded like a nested cursor, and for the same reason: on an update the
3  // callback is handed the changes, so authorId is only there when it changed
4  if (doc.authorId) {
5    const author = await Authors.findOneAsync(doc.authorId, {fields: {name: 1}});
6    doc.authorName = author?.name;
7  }
8});

Editing doc counts anywhere in the callback, before or after an await: the write happens once the callback settles. Returning fields works the same way.

The writes of one document are serialised in the order their events arrived, so added → changed → removed cannot come out reversed even though Meteor's observe multiplexer dispatches live events without waiting for what a callback returns. (Reversed writes are not a subtle failure — ddp-server answers a changed for a document it has not seen with "Could not find element with id X to change" and the subscription dies.) A callback that is not async keeps the Meteor 2 path exactly: the write goes out inline, no queue is allocated.

Main API

to use the following methods you should publish with Meteor.publishRelations instead of Meteor.publish

The package's methods live in their own namespace on this.relations, so they can never collide with something Meteor's Subscription gains later. this in a publication body is the plain subscription, which means this.userId, this.added(), this.ready() and this.onStop() behave exactly as they do in Meteor.publish:

1Meteor.publishRelations('books', function () {
2  this.relations.cursor(Books.find({ownerId: this.userId}));
3
4  return this.ready();
5});

Inside a callback the two swap places: this is that document's methods object, where this.relations is a self-reference — so the same expression keeps working at any depth — and the subscription is reachable as this.sub.

where	`this`	the package API	the subscription
publication body	the `Subscription`	`this.relations`	`this`
inside a callback	that document's methods	`this.relations` (which is `this`)	`this.sub`

Upgrading from 3.x: 4.0 requires Meteor 3 and changes one thing in the API. Earlier versions merged the methods onto the subscription, so this.cursor() and this.ready() sat on one object. Prefix the package's calls in the publication body with relations.; calls inside callbacks need no change, since this is the methods object there either way. Nothing else moves: no body has to become async, no call has to be awaited (see Asynchrony), and the semantics of cursor, join and their callbacks are unchanged. In practice the upgrade of a publication file is this. → this.relations. in the body.

Meteor 2 apps stay on the 3.x line, which keeps the merged API and is feature-frozen.

this.relations.cursor (cursor, collection, callbacks(id, doc, changed))

publishes a cursor, collection is not required

collection is the collection where the cursor will be sent. if not sent, is the default cursor collection name
callbacks is an object with 3 functions (added, changed, removed) or a function that is called when it is added and changed and receive in third parameter a Boolean value that indicates if is changed
If you send callbacks you can use all the methods again and you can edit the document directly (doc.property = 'some') or send it in the return.

Note: when a document changes (update) doc contains only the changes, not the whole document.

That note cuts both ways, and the writing side is the easy one to miss. Editing a field that is not in the update adds it to the update, so the client is told that field changed — and whatever it already held is overwritten:

1this.relations.cursor(Meteor.users.find(), function (id, doc, changed) {
2  // BUG: on any update that does not touch roomId, doc.roomId is undefined, so
3  // this puts 'lobby' into the update and the client loses the real roomId
4  doc.roomId = doc.roomId || 'lobby';
5});

Guard it on the add, or on the field really being part of the update:

1this.relations.cursor(Meteor.users.find(), function (id, doc, changed) {
2  if (!changed || 'roomId' in doc) {
3    doc.roomId = doc.roomId || 'lobby';
4  }
5});

'roomId' in doc also holds when the field was cleared — a removed field arrives in the update with the value undefined — so the default still applies then.

Defaults like this are usually better applied where the data is read, since the publication has to describe a change and not a state.

this.relations.join (Collection, options, name)

It allows you to collect a lot of _ids and then make a single query, only Collection is required.

Collection is the Mongo Collection to be used
options the options parameter in a Collection.find
name the name of a different collection to receive documents there

After creating an instance of this.relations.join you can do the following

1const comments = this.relations.join(Comments, {});
2// options can also be assigned instead of passed, which reads better when the
3// projection is long or when only `name` was worth a constructor argument. Both
4// spellings are the same field, read when the query is built - so any time
5// before send() will do
6comments.options = {fields: {bookId: 1, text: 1}};
7// default query is {_id: {$in: _ids}}
8// if you need to use another field use selector
9comments.selector = function (_ids) {
10  // _ids is always {$in: [...]}, one element or many
11  return {bookId: _ids};
12};
13// Adds a new id to the query. null and undefined are ignored - a `changed`
14// callback is handed the update, so a foreign key it does not carry arrives as
15// undefined, and pushing it is a no-op rather than something to guard against.
16// (0 is a legitimate id and is kept.)
17comments.push(id);
18comments.push(id2, id3, id4);
19// Sends the query to the client. From then on a push that changes what the
20// join holds restarts its query - coalesced, so a burst of them costs one
21// restart, and a push of something already held costs nothing at all. You do
22// not have to worry about reactivity or performance with this method
23comments.send();

Why use this and not this.relations.cursor? because they are just 2 queries

1const comments = this.relations.join(Comments, {});
2comments.selector = _ids => ({bookId: _ids});
3
4this.relations.cursor(Books.find(), function (id, doc) {
5  comments.push(id);
6});
7
8comments.send();

Note: declare a join in the publication body, as above — never inside a callback. push() belongs in callbacks, the join itself does not.

A callback runs again every time its document changes, so a join created inside one is a new instance each time. Each of them registers on that document's handler and none of them is released until the document leaves the result set — at which point every stale instance restarts its own observe and retracts through it. Nothing leaks, but a document that has changed fifty times pays for fifty of them at once, and there is no way for the package to tell that the older ones are finished with.

this.relations.observe / this.relations.observeChanges (cursor, callbacks)

observe or observe changes in a cursor without sending anything to the client, callbacks are the same as those used by meteor

Nonreactive API

The following methods work much like their peers but they are not reactive

this.relations.cursorNonreactive (cursor, collection, callback)

It has 2 differences with this.relations.cursor

callback is only a function that executes when a document is added
you can only use non-reactive methods within the callback

this.relations.joinNonreactive (Collection, options, name)

Is exactly the same as this.relations.join but non reactive

Performance Notes

every method hands back a promise for something with a stop() on it, with one exception: join.send() resolves with nothing when the join has collected no ids, because there is no observer to stop until it has some
registrations in one body are built in parallel, since nothing in the body waits for them — a cold subscribe is therefore faster than on Meteor 2, where a Fiber lined them up one after another
two identical registrations that overlap in time share a single Mongo observe: the package holds a lock per cursor description around the registration, because the multiplexer cache Meteor looks the description up in is filled across an await on Meteor 3 and both would otherwise miss it, build their own driver, and leave one of them orphaned
all cursors are stopped when the publication stop
when the parent cursor is stopped or a document with cursors is removed all related cursors are stopped
all cursors use basic observeChanges as meteor does by default, performance does not come down
if when the callback is re-executes not called again some method (within an If for example), the method continues to run normally, if you re-call method (because the selector is now different) the previous method is replaced with the new

1// For example we have a collection users and each user has a roomId
2// we want to publish the users and their rooms
3this.relations.cursor(Meteor.users.find(), function (id, doc) {
4  // this function is executed on added/changed
5  this.relations.cursor(Rooms.find({_id: doc.roomId}));
6});
7// the previous cursor is good but has a bug, when an user is changed we can't make sure
8// that the roomId is changed and 'doc' only comes with the changes, so roomId is undefined
9// and our Rooms cursor no longer work anymore
10
11// to fix the above problem we need to check the roomId
12this.relations.cursor(Meteor.users.find(), function (id, doc) {
13  if (doc.roomId) {
14    this.relations.cursor(Rooms.find({_id: doc.roomId}));
15  }
16});
17// or we can use an object with 'added' instead of a function
18// this way is better than the above if we are sure that roomId is not going to change
19this.relations.cursor(Meteor.users.find(), {
20  added: function (id, doc) {
21    this.relations.cursor(Rooms.find({_id: doc.roomId}));
22  }
23});

As I said in Quick Start you can do this

1this.relations.cursor(Authors.find(authorId), function (id, doc) {
2  this.relations.cursor(Books.find({authorId: id}), function (id, doc) {
3    this.relations.cursor(Comments.find({bookId: id}));
4  });
5});

but you will find that the publication is becoming increasingly slow, suppose you have 10 books for a given author and every book has 100 reviews, with this method would make the following queries: 1 author + 1 books + 10 comments = 12 queries, for each book found a query is made to find comments which creates a performance issue and publication could take seconds

The solution is to use this.relations.join to join all the comments and send them in a single query, passing from 12 queries to 3 queries for mongo

1const comments = this.relations.join(Comments);
2comments.selector = _ids => ({bookId: _ids});
3
4this.relations.cursor(Authors.find(authorId), function (id, doc) {
5  // We not have to worry about the books cursor because we only have one author
6  this.relations.cursor(Books.find({authorId: id}), function (id, doc) {
7    comments.push(id);
8  });
9});
10
11comments.send();

publications are completed as usual

1// you can do this to finish writing your publication
2this.ready();
3return this.ready();
4return [];
5return [cursor1, cursor2, cursor3];

The cursors you return are published by Meteor itself, exactly as in a plain Meteor.publish, alongside everything the package has already sent — which is the shape to use for the parts of a publication that need no callback and no join at all.

Putting it together

A real publication is usually one list plus everything a row needs to render. Four joined collections here, and the number of queries does not move with the number of books:

1Meteor.publishRelations('bookshelf', function (selector = {}) {
2  // Joins are declared here, in the body, once for the life of the subscription.
3  // Each one is projected: a join without options caches and sends whole
4  // documents, and it is the joins - not the parent cursor - that hold the rows.
5  const authors = this.relations.join(Authors, {fields: {name: 1}});
6  const publishers = this.relations.join(Publishers, {fields: {name: 1, city: 1}});
7  const comments = this.relations.join(Comments, {fields: {bookId: 1, text: 1}});
8  const editors = this.relations.joinNonreactive(Meteor.users, {fields: {profile: 1}});
9
10  // A join whose ids are not the joined documents' own _id needs a selector.
11  comments.selector = _ids => ({bookId: _ids});
12
13  this.relations.cursor(Books.find(selector, {limit: 50, sort: {createdAt: -1}}), function (id, doc) {
14    // Unguarded on purpose. A `changed` callback is handed the update, so on an
15    // edit that leaves authorId alone the key is simply not there - and push()
16    // ignores null and undefined, without so much as allocating anything for the
17    // document. A guard would only be needed around a cursor built from the key.
18    comments.push(id);
19    authors.push(doc.authorId);
20    publishers.push(doc.publisherId);
21    editors.push(doc.editorId);
22  });
23
24  authors.send();
25  publishers.send();
26  comments.send();
27  editors.send();
28
29  return this.ready();
30});

Fifty books or five hundred, Mongo sees one query per cursor plus one per join, and a push that changes what a join holds costs that join one restart — coalesced, so a whole oplog batch is still one. Publications of this shape — a windowed parent cursor and half a dozen joins hanging off it — are what the package is written for, and what it runs in production.

The parts worth copying, in the order they bite:

joins in the body, push in the callbacks — a join constructed inside a callback is a new instance on every re-run
a projection on every join, and a limit on the parent cursor
a guard on every cursor built from a foreign key, because a changed callback is handed the update and the key may not be in it — a push of the same key needs none, push() drops null and undefined itself
send() once per join, after the cursors that feed it — the first materialisation waits for them on its own, so no await is needed anywhere

What is deliberately not in there is a second hop — a field of a joined document that keys yet another collection, say the author's country. A join has no callback to read it from, so that needs a nested cursor on the joined collection, and a nested cursor is per parent row: it is the one part of this shape that grows with the list. When it is worth it anyway, push into the join for the same collection as well (authors.push(doc.authorId) right beside the cursor on Authors) so the join's membership stays a superset of what the cursor publishes — see Limitations.

Limitations

A re-pointed foreign key can keep the old joined document

What a callback pushes is added to what that document already contributes; a re-run never replaces it. Whether a re-pointed foreign key therefore leaves the old joined document behind depends on where the push is:

Both of these react to the same edit — book.authorId going from A1 to A2 — and only the position of the push differs:

1// (1) the callback that reads the key pushes
2this.relations.cursor(Books.find(), function (id, doc) {
3  authors.push(doc.authorId);
4});
5// -> the join holds A1 AND A2
6
7// (2) a cursor that the key rebuilds pushes
8this.relations.cursor(Books.find(), function (id, doc) {
9  if (doc.authorId) {
10    this.relations.cursor(Authors.find({_id: doc.authorId}), function (aid, author) {
11      countries.push(author.countryId);
12    });
13  }
14});
15// -> the join holds A2's country, and A1's is retracted

where the `push` is	what happens on a re-pointed key
the callback that reads the key	the old value stays declared alongside the new one
a nested cursor's callback, when the rebuilt cursor matches something	what the old nested document declared is released and retracted from the client
a nested cursor's callback, when the rebuilt cursor matches nothing	kept, exactly as in the first row

(1) is a deliberate trade, not an oversight. Replacing on every re-run needs the callback to state everything the document contributes, and it cannot: a changed callback is handed the update, not the document, so on any update that does not touch authorId it would declare nothing and drop a valid link. Ending up with one document too many is the better failure, so that is the one the package takes. It costs an extra document on the client and one extra id in the {$in}, both bounded by how many times the contributing documents re-point a key while they are in the result set.

(2) works because the rebuilt cursor re-declares for itself what it still holds, which is a statement (1) has no way to make. The joined documents it no longer declares are retracted, so this is the shape to reach for when a key really does churn — but only where a nested cursor makes sense in the first place. Replacing the join with a nested cursor on the joined collection is not the same move and does not help: stopping an observer sends no removed, so the old document stays on the client from there too.

(3) is where the rebuilt cursor delivered nothing at all, and the package cannot tell "the selector was built from a key this update does not carry" from "the selector is right and nothing matches it". The first means it could not ask and the ids must be kept; the second means they should go. It keeps them, because a document too many beats a document missing — so a key re-pointed at something that does not exist behaves like (1).

Client code normally reads joined documents by the foreign key it finds on the parent, so a superseded one is simply never looked up; where the collection itself is what gets rendered, filter it by the keys the parent documents hold.

Two guarded `this.relations.cursor` calls on the same collection, in one callback

Needs all four of these together, so most publications can stop reading here:

two or more this.relations.cursor calls on the same collection
both inside the same callback
at least one of them guarded by an if
an update that reaches a later call while skipping an earlier one

this.relations.join is not affected at all, however many joins there are on a collection: a join takes its slot key once, when it is constructed in the publication body, and keeps it for the life of the subscription.

A nested cursor, by contrast, is identified by its collection and by the order of the this.relations.cursor calls in the callback - which is what lets a re-run replace its own observer. A guard breaks that ordering when two of them are on the same collection:

1this.relations.cursor(Books.find(), function (id, doc) {
2  if (doc.mainAuthorId) this.relations.cursor(Authors.find({_id: doc.mainAuthorId}));
3  if (doc.editorId) this.relations.cursor(Authors.find({_id: doc.editorId}));   // same collection
4});

On an update carrying editorId but not mainAuthorId the first call is skipped, so the second is now the first Authors cursor of that run and is handed the first one's slot: it stops the main author's observer and leaves its own previous observer running in a slot it no longer claims.

Nothing leaks on the server - the number of observers stays put - but the client does not recover on its own. Each such update leaves one more document sitting in the client's collection with no observer behind it: it will never change again and never be removed, and nothing about it says so. Meanwhile the editor the second cursor stopped claiming keeps its observer, so the client also keeps receiving updates for a document the publication no longer wants.

Two cursors on one collection are fine unguarded, and any number of guarded cursors are fine on different collections. Only the combination bites. Send one of the two under its own name - this.relations.cursor(cursor, 'editors') is a separate slot, and a separate collection on the client - or use the added form from Performance Notes, which does not re-run at all. Removing the guards is not a fix: the callback is handed the update, so the selector would be built from a key that is not in it, and the cursor would be replaced by one matching nothing.

One publication sending the same collection twice

Everything a publication sends goes out under a single subscription handle, and Meteor tracks a published document per handle, not per publisher. So when a join and a cursor of the same publication both send documents of one collection, the first removed from either takes the document away from the client even though the other still matches it — and it comes back only when that other cursor next restarts.

Keep the join's membership a superset of what the cursors publish and it cannot happen, because a retraction then only fires once nothing wants the document any more:

1// Rooms are sent twice here: by the join, and by the nested cursor.
2this.relations.cursor(Meteor.users.find(), function (id, doc) {
3  if (doc.roomId) {
4    rooms.push(doc.roomId);                       // <- keeps the two in step
5    this.relations.cursor(Rooms.find({_id: doc.roomId}), function (id, room) {
6      owners.push(room.ownerId);
7    });
8  }
9});

That is worth checking whenever a publication has both a join and a cursor on one collection: for every such cursor there should be a push into the matching join. Where that is not possible, either give one of them its own name (this.relations.cursor(cursor, 'roomsLookup', callbacks), which sends it to a separate client collection) or use this.relations.observe, which runs callbacks without sending anything at all.

In development the package warns when this bites — once per collection, when a cursor tries to update a document a join has already retracted. Two caveats: the warning reports the consequence, so a document that is retracted and then never changes again produces no warning at all; and it is gated on Meteor.isDevelopment, so a staging build running in production mode stays silent about it.

Testing

The suite has two layers. Run both with one command:

./test.sh          # unit layer only — fast, no side effects
./test.sh --full   # both layers (boots a Meteor test server on $PORT, default 3199)

Unit layer (tests/unit/) loads the server modules into a vm with Meteor stubbed out, so it runs in plain node in milliseconds — no Meteor, no MongoDB. That gives deterministic control over things a DDP client cannot see: what sits in the deferred restart queue, what happens when a release lands in the same tick as a push, how a subscription that is already deactivated behaves, and the overlap guards for two cursors publishing the same collection. Write ordering is provable here rather than in the Tinytest layer: the fake subscription throws the same errors ddp-server does, so a reversed write is a failed assertion and not a line in a log.

node tests/unit/run.js

The test scripts require Node 14.18+ (they import builtins with the node: prefix). ./test.sh checks for that and falls back to the Node bundled with the Meteor tool when the system one is older or missing.

Tinytest layer (tests/*.js) runs against a real Meteor server, a real MongoDB and a real DDP client. It owns what only the real stack can show: DDP message order, and the observer lifecycle — leaks, replacement on restart, and teardown. Observer counts are read from MongoInternals, because a leaked observer is invisible from the client side.

Every test in this package is server-side, so no browser is needed. Instead of the browser reporter (or test-in-console, which drives headless Chrome via puppeteer), tests/headless-driver.js speaks raw DDP to the test server and calls the tinytest/run method directly:

meteor test-packages --release METEOR@3.4.1 --port 3199 ./   # one shell
node tests/headless-driver.js                                # another

It exits non-zero when a test fails, so it can gate CI.

Tinytest has no built-in timeouts, so the tests use a deadline() helper — a regression must report as a failure, not hang the suite.