MailTime
Micro-service package for mail queue, with Server and Client API.
Build on top of nodemailer
package.
Every MailTime
instance can be configured to be a Server or Client.
Main difference of Server from Client - Server handles queue and actually sends email. While Client is only put emails into the queue.
ToC
- How it works?
- Features
- Installation
- Meteor.js Installation: as NPM Package
- Meteor.js Installation: as Atmosphere package
- Usage example
- API
- Custom Templates
- ~92% tests coverage
Main features:
- 👨🔬 ~92% tests coverage;
- 📦 Two simple dependencies, written from scratch for top performance;
- 🏢 Synchronize email queue across multiple servers;
- 💪 Bulletproof design, built-in retries.
How does it work?:
Single point of failure
Issue - classic solution with the single point of failure:
|----------------| |------| |------------------| | Other mailer | ------> | SMTP | ------> | ^_^ Happy user | |----------------| |------| |------------------| The scheme above will work as long as SMTP service is available or connection between your server and SMPT is up. Once network failure occurs or SMTP service is down - users won't be happy |----------------| \ / |------| |------------------| | Other mailer | --X---> | SMTP | ------> | 0_o Disappointed | |----------------| / \ |------| |------------------| ^- email lost in vain Single SMTP solution may work in case of network or other failures As long as MailTime has not received confirmation what email is sent it will keep the letter in the queue and retry to send it again |----------------| / |------| |------------------| | Mail Time | --X---> | SMTP | ------> | ^_^ Happy user | |---^------------| / |------| |------^-----------| \-------------/ ^- We will try later / \- put it back into queue / \----------Once connection is back ------/
Multiple SMTP providers
Backup scheme with multiple SMTP providers
|--------| /--X--| SMTP 1 | / ^ |--------| / \--- Retry with next provider |----------------|/ |--------| |------------------| | Mail Time | ---X--> | SMTP 2 | /->| ^_^ Happy user | |----------------|\ ^ |--------| / |------------------| \ \--- Retry / \ |--------| / \---->| SMTP 3 |--/ |--------|
Cluster issue
Let's say you have an app which is growing fast. At some point, you've decided to create a "Cluster" of servers to balance the load and add durability layer.
Also, your application has scheduled emails, for example, once a day with recent news. While you have had single server emails was sent by some daily interval. So, after you made a "Cluster" of servers - each server has its own timer and going to send a daily email to our user. In such case - users will receive 3 emails, sounds not okay.
Here is how we solve this issue:
|===================THE=CLUSTER===================| |=QUEUE=| |===Mail=Time===| | |----------| |----------| |----------| | | | |=Micro-service=| |--------| | | App | | App | | App | | | | | |-->| SMTP 1 |------\ | | Server 1 | | Server 2 | | Server 3 | | | <-------- | |--------| \ | |-----\----| |----\-----| |----\-----| | | --------> | |-------------| | \---------------\----------------\----------> | | | |--------| | ^_^ | | Each of the "App Server" or "Cluster Node" | | | | |-->| SMTP 2 |-->| Happy users | | runs Mail Time as a Client which only puts | | | | | |--------| |-------------| | emails into the queue. Aside to "App Servers" | | | | | / | We suggest running Mail Time as a Micro-service | | | | | |--------| / | which will be responsible for making sure queue | | | | |-->| SMTP 3 |-----/ | has no duplicates and to actually send emails | | | | | |--------| |=================================================| |=======| |===============|
Features
- Queue - Managed via MongoDB, and will survive server reboots and failures
- Support for multiple server setups - "Cluster", Phusion Passenger instances, Load Balanced solutions, etc.
- Emails concatenation by addressee email - Reduce amount of sent email to single user with concatenation, and avoid mistakenly doubled emails
- When concatenation is enabled - Same emails wouldn't be sent twice, if for any reason, due to bad logic or application failure emails is sent twice or more times - here is solution to solve this annoying behavior
- Balancing for multiple nodemailer's transports, two modes -
backup
andbalancing
. Most useful feature - allows to reduce the cost of SMTP services and add durability. So, if any of used transports are failing to send an email it will switch to next one - Sending retries for network and other failures
- Template support with Mustache-like placeholders
Installation
If you're working on Server functionality - first you will need nodemailer
, although this package is meant to be used with nodemailer
, it's not added as the dependency, as it not needed by Client, and you're free to choose nodemailer
's version to fit your needs:
npm install --save nodemailer
Install MailTime package:
# for node@>=8.9.0 npm install --save mail-time # for node@<8.9.0 npm install --save mail-time@=0.1.7
Basic usage
Require package:
1const MailTime = require('mail-time');
Create nodemailer's transports (see nodemailer docs):
1const transports = []; 2const nodemailer = require('nodemailer'); 3 4// Use DIRECT transport 5// To enable sending email from localhost 6// install "nodemailer-direct-transport" NPM package: 7const directTransport = require('nodemailer-direct-transport'); 8const directTransportOpts = { 9 pool: false, 10 direct: true, 11 name: 'mail.example.com', 12 from: 'no-reply@example.com', 13}; 14transports.push(nodemailer.createTransport(directTransport(directTransportOpts))); 15// IMPORTANT: Copy-paste passed options from directTransport() to 16// transport's "options" property, to make sure it's available to MailTime package: 17transports[0].options = directTransportOpts; 18 19// Private SMTP 20transports.push(nodemailer.createTransport({ 21 host: 'smtp.example.com', 22 from: 'no-reply@example.com', 23 auth: { 24 user: 'no-reply', 25 pass: 'xxx' 26 }, 27})); 28 29// Google Apps SMTP 30transports.push(nodemailer.createTransport({ 31 host: 'smtp.gmail.com', 32 from: 'no-reply@mail.example.com', 33 auth: { 34 user: 'no-reply@mail.example.com', 35 pass: 'xxx' 36 }, 37})); 38 39// Mailing service (SparkPost as example) 40transports.push(nodemailer.createTransport({ 41 host: 'smtp.sparkpostmail.com', 42 port: 587, 43 from: 'no-reply@mail2.example.com', 44 auth: { 45 user: 'SMTP_Injection', 46 pass: 'xxx' 47 }, 48}));
Create mail-time
Server, it is able to send and add emails to the queue.
We will need connect to MongoDB first:
1const MongoClient = require('mongodb').MongoClient; 2const MailTime = require('mail-time'); 3const dbName = 'DatabaseName'; 4 5// We're using environment variable MONGO_URL 6// to store connection string to MongoDB 7// example: "MONGO_URL=mongodb://127.0.0.1:27017/myapp node mail-micro-service.js" 8MongoClient.connect(process.env.MONGO_URL, (error, client) => { 9 const db = client.db(dbName); 10 11 const mailQueue = new MailTime({ 12 db, // MongoDB 13 type: 'server', 14 strategy: 'balancer', // Transports will be used in round robin chain 15 transports, 16 from(transport) { 17 // To pass spam-filters `from` field should be correctly set 18 // for each transport, check `transport` object for more options 19 return '"Awesome App" <' + transport.options.from + '>'; 20 }, 21 concatEmails: true, // Concatenate emails to the same addressee 22 concatDelimiter: '<h1>{{{subject}}}</h1>', // Start each concatenated email with it's own subject 23 template: MailTime.Template // Use default template 24 }); 25});
Create the Client to add emails to queue from other application units, like UI unit:
1const MongoClient = require('mongodb').MongoClient; 2const MailTime = require('mail-time'); 3const dbName = 'DatabaseName'; 4 5MongoClient.connect(process.env.MONGO_URL, (error, client) => { 6 const db = client.db(dbName); 7 8 const mailQueue = new MailTime({ 9 db, 10 type: 'client', 11 strategy: 'balancer', // Transports will be used in round robin chain 12 concatEmails: true // Concatenate emails to the same address 13 }); 14});
Send email:
1mailQueue.sendMail({ 2 to: 'user@gmail.com', 3 subject: 'You\'ve got an email!', 4 text: 'Plain text message', 5 html: '<h1>HTML</h1><p>Styled message</p>' 6});
Meteor.js usage:
Meteor.js Installation:
Installation & Import (via NPM):
Install NPM MailTime package:
meteor npm install --save mail-time
ES6 Import:
1import MailTime from 'mail-time';
Installation & Import (via Atmosphere):
Install Atmosphere ostrio:mailer package:
meteor add ostrio:mailer
ES6 Import:
1import MailTime from 'meteor/ostrio:mailer';
Usage:
1import { MongoInternals } from 'meteor/mongo'; 2 3import MailTime from 'mail-time'; 4import nodemailer from 'nodemailer'; 5// Use DIRECT transport 6// To enable sending email from localhost 7// install "nodemailer-direct-transport" NPM package: 8import directTransport from 'nodemailer-direct-transport'; 9 10const transports = []; 11const directTransportOpts = { 12 pool: false, 13 direct: true, 14 name: 'mail.example.com', 15 from: 'no-reply@example.com', 16}; 17transports.push(nodemailer.createTransport(directTransport(directTransportOpts))); 18// IMPORTANT: Copy-paste passed options from directTransport() to 19// transport's "options" property, to make sure it's available to MailTime package: 20transports[0].options = directTransportOpts; 21 22//////////////////////// 23// For more transports example see sections above and read nodemailer's docs 24//////////////////////// 25 26const mailQueue = new MailTime({ 27 db: MongoInternals.defaultRemoteCollectionDriver().mongo.db, // MongoDB 28 transports, 29 from(transport) { 30 // To pass spam-filters `from` field should be correctly set 31 // for each transport, check `transport` object for more options 32 return '"Awesome App" <' + transport.options.from + '>'; 33 } 34});
API
new MailTime(opts)
constructor
opts
{Object} - Configuration objectopts.db
{Db} - [Required] Mongo'sDb
instance. For example returned in callback ofMongoClient.connect()
opts.type
{String} - [Optional]client
orserver
, default -server
opts.from
{Function} - [Optional] A function which returns String offrom
field, format:"MyApp" <user@example.com>
opts.transports
{Array} - [Optional] An array ofnodemailer
's transports, returned fromnodemailer.createTransport({})
opts.strategy
{String} - [Optional]backup
orbalancer
, default -backup
. If set tobackup
, first transport will be used unless failed to sendfailsToNext
times. If set tobalancer
- transports will be used equally in round robin chainopts.failsToNext
{Number} - [Optional] After how many failed "send attempts" switch to next transport, applied only forbackup
strategy, default -4
opts.prefix
{String} - [Optional] Use unique prefixes to create multipleMailTime
instances on same MongoDBopts.maxTries
{Number} - [Optional] How many times resend failed emails, default -60
opts.interval
{Number} - [Optional] Interval in seconds between send re-tries, default -60
opts.zombieTime
{Number} - [Optional] Time in milliseconds, after this period - pending email will be interpreted as "zombie". This parameter allows to rescue pending email from "zombie mode" in case when: server was rebooted, exception during runtime was thrown, or caused by bad logic, default -32786
. This option is used by package itself and passed directly toJoSk
packageopts.keepHistory
{Boolean} - [Optional] By default sent emails not stored in the database. Set{ keepHistory: true }
to keep queue task as it is in the database, default -false
opts.concatEmails
{Boolean} - [Optional] Concatenate email byto
field, default -false
opts.concatSubject
{String} - [Optional] Email subject used in concatenated email, default -Multiple notifications
opts.concatDelimiter
{String} - [Optional] HTML or plain string delimiter used between concatenated email, default -<hr>
opts.concatThrottling
{Number} - [Optional] Time in seconds while emails are waiting to be concatenated, default -60
opts.revolvingInterval
{Number} - [Optional] Interval in milliseconds in between queue checks, default -256
. Recommended value — betweenopts.minRevolvingDelay
andopts.maxRevolvingDelay
opts.minRevolvingDelay
{Number} - [Optional] Minimum revolving delay — the minimum delay between tasks executions in milliseconds, default -64
. This option is passed directly toJoSk
packageopts.maxRevolvingDelay
{Number} - [Optional] Maximum revolving delay — the maximum delay between tasks executions in milliseconds, default -256
. This option is passed directly toJoSk
packageopts.template
{String} - [Optional] Mustache-like template, default -{{{html}}}
, all options passed tosendMail
is available in Template, liketo
,subject
,text
,html
or any other custom option. Use{{opt}}
for string placeholders and{{{opt}}}
for html placeholders
sendMail(opts [, callback])
- Alias -
send()
opts
{Object} - Configuration objectopts.sendAt
{Date} - When email should be sent, default -new Date()
use with caution on multi-server setup at different location with the different time-zonesopts.template
- Email specific template, this will override default template passed toMailTime
constructoropts.concatSubject
- Email specific concatenation subject, this will override default concatenation subject passed toMailTime
constructoropts[key]
{Mix} - Other custom and NodeMailer specific options, liketext
,html
andto
, see more here. Noteattachments
should work only viapath
, and file must exists on all micro-services serverscallback
{Function} - Callback called after the email was sent or failed to be sent. Do not use on multi-server setup
static MailTime.Template
Simple and bulletproof HTML template, see its source. Usage:
1const MailTime = require('mail-time'); 2// Make it default 3const mailQueue = new MailTime({ 4 db: db, // MongoDB 5 /* .. */ 6 template: MailTime.Template 7}); 8 9// For single letter 10mailQueue.sendMail({ 11 to: 'user@gmail.com', 12 /* .. */ 13 template: MailTime.Template 14});
Template Example
1mailQueue.sendMail({ 2 to: 'user@gmail.com', 3 userName: 'Mike', 4 subject: 'Sign up confirmation', 5 text: 'Hello {{userName}}, \r\n Thank you for registration \r\n Your login: {{to}}', 6 html: '<div style="text-align: center"><h1>Hello {{userName}}</h1><p><ul><li>Thank you for registration</li><li>Your login: {{to}}</li></ul></p></div>', 7 template: '<body>{{{html}}}</body>' 8});
Testing
- Clone this package
- In Terminal (Console) go to directory where package is cloned
- Then run:
Test NPM package:
# Before run tests make sure NODE_ENV === development # Install NPM dependencies npm install --save-dev # Before run tests you need to have running MongoDB DEBUG="true" EMAIL_DOMAIN="example.com" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test # Be patient, tests are taking around 2 mins
Test Atmosphere (meteor.js) package:
# Default EMAIL_DOMAIN="example.com" meteor test-packages ./ --driver-package=meteortesting:mocha # With custom port DEBUG="true" EMAIL_DOMAIN="example.com" meteor test-packages ./ --driver-package=meteortesting:mocha --port 8888 # With local MongoDB and custom port DEBUG="true" EMAIL_DOMAIN="example.com" MONGO_URL="mongodb://127.0.0.1:27017/meteor-mail-time-test-001" meteor test-packages ./ --driver-package=meteortesting:mocha --port 8888 # Be patient, tests are taking around 2 mins
Support this project:
- Support via PayPal — support my open source contributions once or on regular basis
- Use ostr.io — Monitoring, Analytics, WebSec, Web-CRON and Pre-rendering for a website