MailTime
"Mail-Time" is a micro-service package for mail queue, with Server and Client APIs. Build on top of the nodemailer
package. Mail-Time made for single-server and horizontally scaled multi-server setups in mind.
Every MailTime
instance can have type
configured as Server or Client.
The main difference between Server and Client type
is that the Server handles the queue and sends email. While the Client only adds emails into the queue.
ToC
- How it works?
- Features
- Installation
- Meteor.js usage
- Usage example
- API
- Custom Templates
- ~92% tests coverage
Main features:
- 👨🔬 ~92% tests coverage;
- 📦 Two simple dependencies, written from scratch for top performance;
- 🏢 Synchronize email queue across multiple (horizontally scaled) servers;
- 💪 Bulletproof design, built-in retries.
How does it work?
Redundant solution for email transmission.
Single point of failure
Issue - mitigate a single point of failure:
|----------------| |------| |------------------| | Other mailer | ------> | SMTP | ------> | ^_^ Happy user | |----------------| |------| |------------------| The scheme above will work as long as SMTP service is available or connection between your server and SMPT is up. Once network failure occurs or SMTP service is down - users won't be happy |----------------| \ / |------| |------------------| | Other mailer | --X---> | SMTP | ------> | 0_o Disappointed | |----------------| / \ |------| |------------------| ^- email lost in vain Single SMTP solution may work in case of network or other failures As long as MailTime has not received confirmation what email is sent it will keep the letter in the queue and retry to send it again |----------------| / |------| |------------------| | Mail Time | --X---> | SMTP | ------> | ^_^ Happy user | |---^------------| / |------| |------^-----------| \-------------/ ^- We will try later / \- put it back into queue / \----------Once connection is back ------/
Multiple SMTP providers
Rotate or backup email transports by using multiple SMTP providers
|--------| /--X--| SMTP 1 | / ^ |--------| / \--- Retry with next provider |----------------|/ |--------| |------------------| | Mail Time | ---X--> | SMTP 2 | /->| ^_^ Happy user | |----------------|\ ^ |--------| / |------------------| \ \--- Retry / \ |--------| / \---->| SMTP 3 |--/ |--------|
Sending emails from cluster of servers
It is common to have horizontally scaled "Cluster" of servers for load-balancing and for durability.
Most modern application has scheduled or recurring emails. For example, once a day — with recent news and updates. It won't be an issue with a single server setup — the server would send emails at a daily interval via timer or CRON. But in "Cluster" implementation — each server will attempt to send the same email. MailTime built to avoid sending the same email multiple times to a user from horizontally scaled applications.
For the maximum durability and agility each Application Server can run MailTime in the "Server" mode:
|===================THE=CLUSTER===================| |=QUEUE=| | |----------| |----------| |----------| | | | |--------| | | App | | App | | App | | | |-->| SMTP 1 |------\ | | Server 1 | | Server 2 | | Server 3 | | | | |--------| \ | |-----\----| |----\-----| |----\-----| | | | |-------------| | \---------------\----------------\----------> | |--------| | ^_^ | | | | |-->| SMTP 2 |-->| Happy users | | Each "App Server" or "Cluster Node" | | | |--------| |-------------| | runs MailTime as a "Server" | | | / | for the maximum durability | | | |--------| / | | | |-->| SMTP 3 |-----/ | | | | |--------| |=================================================| |=======|
To split roles MailTime can run on a dedicated machine as micro-service. This case is great for private email servers with implemented authentication via rDNS and PTR records:
|===================THE=CLUSTER===================| |=QUEUE=| |===Mail=Time===| | |----------| |----------| |----------| | | | | | |--------| | | App | | App | | App | | | | | Micro-service |-->| SMTP 1 |------\ | | Server 1 | | Server 2 | | Server 3 | | | | | running | |--------| \ | |-----\----| |----\-----| |----\-----| | | | | MailTime as | |-------------| | \---------------\----------------\----------> | | "Server" only | |--------| | ^_^ | | | | | | sending |-->| SMTP 2 |-->| Happy users | | Each "App Server" runs MailTime as | | | | emails | |--------| |-------------| | a "Client" only placing emails to the queue. | | <-------- | / | | | --------> | |--------| / | | | | | |-->| SMTP 3 |-----/ | | | | | | |--------| |=================================================| |=======| |===============|
Features
- Queue - Managed via MongoDB, will survive server reboots and failures
- Support for multiple server setups - "Cluster", Phusion Passenger instances, Load Balanced solutions, etc.
- Emails concatenation by addressee email - Reduce amount of sent emails to a single user with concatenation, and avoid mistakenly doubled emails
- When concatenation is enabled - Same emails won't be sent twice, if for any reason, due to bad logic or application failure emails are sent twice or more times - this is solution to solve this annoying behavior
- Balancing for multiple nodemailer's transports, two modes -
backup
andbalancing
. This is the most useful feature — allowing to reduce the cost of SMTP services and add extra layer of durability. If one transport failing to send an emailmail-time
will switch to the next one - Sending retries for network and other failures
- Templating support with Mustache-like placeholders
Installation
To implement Server functionality — as a first step install nodemailer
, although this package meant to be used with nodemailer
, it's not added as the dependency, as nodemailer
not needed by Client, and you're free to choose nodemailer
's version to fit your project needs:
npm install --save nodemailer
Install MailTime package:
# for node@>=14.20.0 npm install --save mail-time # for node@<14.20.0 npm install --save mail-time@=1.3.4 # for node@<8.9.0 npm install --save mail-time@=0.1.7
Basic usage
Require package:
1// import as ES Module 2import MailTime from 'mail-time'; 3 4// requires as CommonJS 5const MailTime = require('mail-time');
Create nodemailer's transports, for details see nodemailer
docs:
1import nodemailer from 'nodemailer'; 2// Use DIRECT transport 3// and enable sending email from localhost 4// install "nodemailer-direct-transport" NPM package: 5import directTransport from 'nodemailer-direct-transport'; 6 7const transports = []; 8const directTransportOpts = { 9 pool: false, 10 direct: true, 11 name: 'mail.example.com', 12 from: 'no-reply@example.com', 13}; 14transports.push(nodemailer.createTransport(directTransport(directTransportOpts))); 15// IMPORTANT: Add `.options` to a newly created transport, 16// this is necessary to make sure options are available to MailTime package: 17transports[0].options = directTransportOpts; 18 19// Private SMTP 20transports.push(nodemailer.createTransport({ 21 host: 'smtp.example.com', 22 from: 'no-reply@example.com', 23 auth: { 24 user: 'no-reply', 25 pass: 'xxx' 26 }, 27})); 28 29// Google Apps SMTP 30transports.push(nodemailer.createTransport({ 31 host: 'smtp.gmail.com', 32 from: 'no-reply@mail.example.com', 33 auth: { 34 user: 'no-reply@mail.example.com', 35 pass: 'xxx' 36 }, 37})); 38 39// Mailing service (SparkPost as example) 40transports.push(nodemailer.createTransport({ 41 host: 'smtp.sparkpostmail.com', 42 port: 587, 43 from: 'no-reply@mail2.example.com', 44 auth: { 45 user: 'SMTP_Injection', 46 pass: 'xxx' 47 }, 48}));
As the next step initiate mail-time
in the Server mode, it will be able to send and add emails to the queue. Connecting to a MongoDB before initiating new MailTime
instance:
1const MailTime = require('mail-time'); 2const MongoClient = require('mongodb').MongoClient; 3 4const dbName = 'databaseName'; 5 6// Use MONGO_URL environment variable to store connection string to MongoDB 7// example: "MONGO_URL=mongodb://127.0.0.1:27017/myapp node mail-micro-service.js" 8MongoClient.connect(process.env.MONGO_URL, (error, client) => { 9 const db = client.db(dbName); 10 11 const mailQueue = new MailTime({ 12 db, // MongoDB 13 type: 'server', 14 strategy: 'balancer', // Transports will be used in round robin chain 15 transports, 16 from(transport) { 17 // To pass spam-filters `from` field should be correctly set 18 // for each transport, check `transport` object for more options 19 return `"Awesome App" <${transport.options.from}>`; 20 }, 21 concatEmails: true, // Concatenate emails to the same addressee 22 concatDelimiter: '<h1>{{{subject}}}</h1>', // Start each concatenated email with it's own subject 23 template: MailTime.Template // Use default template 24 }); 25});
Only one MailTime
Server instance required to send email. In the other parts of an app (like UI units or in sub-apps) use mail-time
in the Client mode to add emails to queue:
1const MailTime = require('mail-time'); 2const MongoClient = require('mongodb').MongoClient; 3 4const dbName = 'databaseName'; 5 6MongoClient.connect(process.env.MONGO_URL, (error, client) => { 7 const db = client.db(dbName); 8 9 const mailQueue = new MailTime({ 10 db, 11 type: 'client', 12 strategy: 'balancer', // Transports will be used in round robin chain 13 concatEmails: true // Concatenate emails to the same address 14 }); 15});
Send email example:
1mailQueue.sendMail({ 2 to: 'user@gmail.com', 3 subject: 'You\'ve got an email!', 4 text: 'Plain text message', 5 html: '<h1>HTML</h1><p>Styled message</p>' 6});
Two MailTime
instances usage example
Create two MailTime
instances with different settings.
1// CREATE mailQueue FOR NON-URGENT EMAILS WHICH IS OKAY TO CONCATENATE 2const mailQueue = new MailTime({ 3 db: db, 4 interval: 35, 5 strategy: 'backup', 6 failsToNext: 1, 7 concatEmails: true, 8 concatThrottling: 16, 9 zombieTime: 120000 10}); 11 12// CREATE mailInstantQueue FOR TRANSACTIONAL EMAILS AND ALERTS 13const mailInstantQueue = new MailTime({ 14 db: db, 15 prefix: 'instant', 16 interval: 2, 17 strategy: 'backup', 18 failsToNext: 1, 19 concatEmails: false, 20 zombieTime: 20000 21}); 22 23mailQueue.sendMail({ 24 to: 'user@gmail.com', 25 subject: 'You\'ve got an email!', 26 text: 'Plain text message', 27 html: '<h1>HTML</h1><p>Styled message</p>' 28}); 29 30mailInstantQueue.sendMail({ 31 to: 'user@gmail.com', 32 subject: 'Sign in request', 33 text: 'Your OTP login code: xxxx:', 34 html: '<h1>Code:</h1><code>XXXX</code>' 35});
Passing variables to the template
All options passed to the .sendMail()
method is available inside text
, html
, and global templates
1const templates = { 2 global: '<html xmlns="http://www.w3.org/1999/xhtml"><head><title>{{subject}}</title></head><body>{{{html}}}<footer>Message sent to @{{username}} user ({{to}})</footer></body></html>', 3 signInCode: { 4 text: 'Hello @{{username}}! Here\'s your login code: {{code}}', 5 html: `<h1>Sign-in request</h1><p>Hello @{{username}}! <p>Copy your login code below:</p> <pre><code>{{code}}</code></pre>` 6 } 7}; 8 9const mailQueue = new MailTime({ 10 db: db, 11 template: templates.global 12}); 13 14mailQueue.sendMail({ 15 to: 'user@gmail.com', 16 subject: 'Sign-in request', 17 username: 'johndoe', 18 code: 'XXXXX-YY', 19 text: templates.signInCode.text, 20 html: templates.signInCode.html 21});
API
All available constructor options and .sendMail()
method API overview
new MailTime(opts)
constructor
opts
{Object} - Configuration objectopts.db
{Db} - [Required] Mongo'sDb
instance. For example returned in callback ofMongoClient.connect()
opts.type
{String} - [Optional]client
orserver
, default -server
opts.from
{Function} - [Optional] A function which returns String offrom
field, format:"MyApp" <user@example.com>
opts.transports
{Array} - [Optional] An array ofnodemailer
's transports, returned fromnodemailer.createTransport({})
opts.strategy
{String} - [Optional]backup
orbalancer
, default -backup
. If set tobackup
, first transport will be used unless failed to sendfailsToNext
times. If set tobalancer
- transports will be used equally in round robin chainopts.failsToNext
{Number} - [Optional] After how many failed "send attempts" switch to next transport, applied only forbackup
strategy, default -4
opts.prefix
{String} - [Optional] Use unique prefixes to create multipleMailTime
instances on same MongoDBopts.maxTries
{Number} - [Optional] How many times resend failed emails, default -60
opts.interval
{Number} - [Optional] Interval in seconds between send re-tries, default -60
opts.zombieTime
{Number} - [Optional] Time in milliseconds, after this period - pending email will be interpreted as "zombie". This parameter allows to rescue pending email from "zombie mode" in case when: server was rebooted, exception during runtime was thrown, or caused by bad logic, default -32786
. This option is used by package itself and passed directly toJoSk
packageopts.keepHistory
{Boolean} - [Optional] By default sent emails not stored in the database. Set{ keepHistory: true }
to keep queue task as it is in the database, default -false
opts.concatEmails
{Boolean} - [Optional] Concatenate email byto
field, default -false
opts.concatSubject
{String} - [Optional] Email subject used in concatenated email, default -Multiple notifications
opts.concatDelimiter
{String} - [Optional] HTML or plain string delimiter used between concatenated email, default -<hr>
opts.concatThrottling
{Number} - [Optional] Time in seconds while emails are waiting to be concatenated, default -60
opts.revolvingInterval
{Number} - [Optional] Interval in milliseconds in between queue checks, default -256
. Recommended value — betweenopts.minRevolvingDelay
andopts.maxRevolvingDelay
opts.minRevolvingDelay
{Number} - [Optional] Minimum revolving delay — the minimum delay between tasks executions in milliseconds, default -64
. This option is passed directly toJoSk
packageopts.maxRevolvingDelay
{Number} - [Optional] Maximum revolving delay — the maximum delay between tasks executions in milliseconds, default -256
. This option is passed directly toJoSk
packageopts.template
{String} - [Optional] Mustache-like template, default -{{{html}}}
, all options passed tosendMail
is available in Template, liketo
,subject
,text
,html
or any other custom option. Use{{opt}}
for string placeholders and{{{opt}}}
for html placeholders
sendMail(opts [, callback])
- Alias -
send()
opts
{Object} - Configuration objectopts.sendAt
{Date} - When email should be sent, default -new Date()
use with caution on multi-server setup at different location with the different time-zonesopts.template
- Email specific template, this will override default template passed toMailTime
constructoropts.concatSubject
- Email specific concatenation subject, this will override default concatenation subject passed toMailTime
constructoropts[key]
{Mix} - Other custom and NodeMailer specific options, liketext
,html
andto
, learn more here. Noteattachments
should work only viapath
, and file must exists on all micro-services serverscallback
{Function} - Callback called after the email was sent or failed to be sent. Do not use on multi-server setup
static MailTime.Template
Simple and bulletproof HTML template, see its source. Usage example:
1const MailTime = require('mail-time'); 2// Make it default 3const mailQueue = new MailTime({ 4 db: db, // MongoDB 5 /* .. */ 6 template: MailTime.Template 7}); 8 9// For single letter 10mailQueue.sendMail({ 11 to: 'user@gmail.com', 12 /* .. */ 13 template: MailTime.Template 14});
Template Example
Pass custom template via template
property to .sendMail()
method
1mailQueue.sendMail({ 2 to: 'user@gmail.com', 3 userName: 'Mike', 4 subject: 'Sign up confirmation', 5 text: 'Hello {{userName}}, \r\n Thank you for registration \r\n Your login: {{to}}', 6 html: '<div style="text-align: center"><h1>Hello {{userName}}</h1><p><ul><li>Thank you for registration</li><li>Your login: {{to}}</li></ul></p></div>', 7 template: '<body>{{{html}}}</body>' 8});
Testing
- Clone this package
- In Terminal (Console) go to directory where package is cloned
- Then run:
Test NPM package:
# Before run tests make sure NODE_ENV === development # Install NPM dependencies npm install --save-dev # Before run tests you need to have running MongoDB DEBUG="true" EMAIL_DOMAIN="example.com" MONGO_URL="mongodb://127.0.0.1:27017/npm-mail-time-test-001" npm test # Be patient, tests are taking around 2 mins
Support this project:
- Upload and share files using ☄️ meteor-files.com — Continue interrupted file uploads without losing any progress. There is nothing that will stop Meteor from delivering your file to the desired destination
- Use ▲ ostr.io for Server Monitoring, Web Analytics, WebSec, Web-CRON and SEO Pre-rendering of a website
- Star on GitHub
- Star on NPM
- Star on Atmosphere
- Sponsor maintainer via GitHub — support open source with one-time contribution or on a regular basis
- Sponsor veliovgroup via GitHub — support company behind this package
- Support via PayPal — support our open source contributions