Sending Email from FileMaker Using SparkPost
By Daniel Wood, 19 March 2016
A bit of background
Back in 2014 we published this article about a new service at the time called Mandrill. Mandrill is an email web service that you can use to send thousands of emails per month. The great thing about Mandrill was that although it had paid subscription options, their free offering was perfect for most FileMaker solutions required to send email. Because email was sent through an API, it also avoids many common SMTP issues that developers come across when trying to setup emailing from FileMaker.
Recently however, Mandrill has announced that it is incorporating Mandrill into its Mailchimp service, and it will become a paid add-on to MailChimp accounts. This change will take effect from April 27th, 2016 and will affect all existing Mandrill users, and any FileMaker integrations that rely on it.
How does this affect me?
If you are not a Mandrill user, there is no effect, and you can read this post at your leisure. If you do use Mandrill in your FileMaker solution you have cause for taking some urgent action. Here is what is changing:
Mandrill users will be required to have a paid monthly MailChimp account
The separate service known as Mandrill will cease to be its own standalone service and will become incorporated into MailChimp by the end of April. To begin this process, MailChimp is requiring you sign up for an account if not already, and link your existing Mandrill account to it.
This in itself is not really a difficult task, sign up is very easy at mailchimp.com and linking your account is easy too. You have until 27th of April to do this. The following is from Mandrill:
The MailChimp team built Mandrill in 2012 as a transactional email tool. It was a startup within MailChimp that functioned as a completely separate product. Mandrill is now becoming an optional add-on to paid MailChimp accounts. This is a strategic change we’re making to close the gap between the two products. MailChimp is designed for marketing, and Mandrill is designed for transactional messages. Each is powerful on its own, and offering Mandrill as an add-on makes MailChimp even more powerful for our small business and e-commerce customers
The real gotcha with the change is that your MailChimp account will be required to be a paid monthly account by 27 April. Any free accounts will no longer be valid once your Mailchimp and Mandrill accounts are merged.
Pricing is set to change and is not yet finalised on the website. The blog article on the subject details that you will pay a single consolidated monthly fee for both Mailchimp and Mandrill, and Mandrill credits will be sold in blocks of 25,000 emails at $20 per month starting price.
Your sending domains will need to be verified
Mandrill allowed you to send emails from any sending domain. A sending domain is the domain name that comes after the @ part of your email address. This made setup easy but it did mean that the service could be abused by spammers and spoofed email addresses acting as senders.
MailChimp will require that all senders verify their domains. This is actually a good thing and not too difficult to do. Normally verifying your sending domain is done by adding a record into either your SPF or DKIM entries in your domain setup. Your domain provider will be able to assist and provide documentation on how you can do this.
One thing to note here that is important is that you'll need your own verified sending domain and you will be unable to use public domains such as Gmail, Hotmail or Yahoo. For users of free email services such as these it may mean you are out of luck.
SparkPost to the Rescue!
This is where SparkPost comes in. SparkPost is a new web service sending emails much like Mandrill, but SparkPost has a free email plan and is a great alternative for those who wish to not pay for Mandrill. SparkPost is recommended by Mandrill as the alternative for those who wish not to move to MailChimp, that's a good endorsement!
Better still, SparkPost offer a Mandrill Migration Guide which will help anyone with the transition.
SparkPost offers a free 100,00 emails a month plan (capped).
They have a great and fully complete API and excellent developer support.
A couple of Points about SparkPost
Sparkpost will require you to verify your sending domain using SPF/DKIM records, or verification through email. This is the same verification that Mandrill is enforcing, so Gmail/Hotmail/Yahoo users will be out of luck, but those with a legitimate sending domain will have no problem with this.
SparkPost can send emails of up to 20mb in size (with attachments) which should pose no issues for most people.
SparkPost also has a very in depth web based dashboard interface for tracking all mail sent, much like the Mandrill dashboard, and a number of features of Sparkpost are very similar to Mandrill.
Even JSON for formulating an API request is very similar to Mandrill so you should have no trouble with the transition.
Migrating to SparkPost
To give some information about the migration from Mandrill to SparkPost, we have used our old Mandrill example file as the basis, and have modified it to work with SparkPost. You can download the old file below.
The new file is also available here. Below we'll outline some of the important differences between the two that you'll need to be aware of when migrating.
Authentication Differences
When authenticating with Mandrill, your API key was passed as part of your JSON request posted to the API endpoint.
With SparkPost however you must first authenticate using your API key by way of setting a custom POST header prior to sending the JSON request. You also need to specify the content type of your request. SparkPost will use XML by default so you must specify whether your request is to be in JSON or XML.
To do this, we use the BaseElements plugin from Goya once again. This contains functions for setting your custom headers.
BE_HTTP_Set_Custom_Header ( "Authorization" ; <APIKEY> )
BE_HTTP_Set_Custom_Header ( "Content-Type" ; "application/json" )
Using these two functions, you can authenticate and specify JSON. These can be combined into a single "Let" statement when posting your request, they need not be specified in separate script steps, or as separate variables, just as long as they're defined prior to posting your request.
Request Differences
While the JSON formatting is very similar to Mandrill, SparkPost has slightly different naming for certain parts of the request such as recipients, but overall it is incredibly similar in how a request is formulated. The demo file contains an equivalent example to the Mandrill demo file, in that it is a template for sending an email to a single recipient with attachments and rich text.
The SparkPost API documentation is great and gives you all the information you'll need to customise the request to suit your needs. The key parts of the documentation around sending emails for you will be the "Transmissions" area and "Recipients" areas. The API can do all kinds of things such as manipulation of your settings, profile, recipient lists, webhooks etc, but a lot of that is not required for simply sending email.
Above is a basic template for sending a SparkPost request. There are only minor changes to that of a Mandrill request, mainly around the structuring of recipients into an array. The attachment code is basically the same.
Parsing the Result Returned
The format of the result returned by the API is different to that of Mandrill and so you will need to modify any functions you use to parse the response. SparkPost responses come in one of two varieties. The first is an error that will be returned along with an error code and message describing the error. You will normally see this if you have an improperly formed JSON request, or are missing vital information in your request.
If the JSON request is correct and an email is sent, you will receive a response that gives information about the number of recipients sent to successfully, or number that failed, along with the ID of the request.
An example of an error message:
{ "errors": [ { "message": "required field is missing", "description": "content object or template_id required", "code": "1400" } ] }
This is a JSON error object containing an array of errors. Each object within the array contains 3 attributes about the error. "Message" is the name of the error, and "Description" will give you more information about what the problem is. "Code" is an error code you can use to look up more information on the error in the API documentation. If you have multiple errors you will receive this in an array.
For a successful response, the format is:
{ "results": { "total_rejected_recipients": 0, "total_accepted_recipients": 1, "id": "84290312071045307" } }
This is a results object that contains the total number of rejected recipients, total number of accepted, and the SparkPost ID.
In our demo file we have included a very simplistic parsing function that will transform this into a return delimited list for easier FileMaker parsing. You can nominate which attribute to return, or passing an empty parameter to the function will return the full result. This is similar to the original parsing function in the Mandrill example file.
Rules around what is and is not Spam
If you find that your emails are not being delivered by SparkPost, it may be due to the rules around what SparkPost deems as suspicious or spam emails, and these may be different to Mandrill. SparkPost is quite strict around checking emails for potential spam. Subject lines, email content, recipients etc all contribute to whether the email will bounce or be delivered.
Originally, our demo file had a test email with the subject line "Hello There!". While this worked fine in Mandrill, SparkPost considered this email spam. We have had to change the subject line for this example file as a result. We recommend testing sending various emails after you switch to SparkPost to ensure they are successfully delivered, especially if you use preset templates.
Sending Email using the "Insert from URL" script step
In the Mandrill example file, reader and developer Michael Sloper found a means to send requests to Mandrill in a URL format which allows you to use the Insert from URL script step. this was great because it removed the need for a plugin.
We have been unable to find a suitable means to achieve this using SparkPost - to be fair we haven't really tried very hard! We welcome anyone who can find a suitable means to integrate with SparkPost using the Insert from URL script step!
In Conclusion
SparkPost is our preferred alternative to Mandrill because it offers the same free plans that Mandrill used to offer, and is very easy to switch to. Our Mandrill article was our most successful ever in terms of reader feedback, and we hope that this article prompts a similar level of discussion and sharing of ideas when it comes to SparkPost integration.
Good luck!
An important note on the API Key and Account used in the Demo file
Because SparkPost requires a valid sending domain for each account, the provided example file has been setup with a preset key on a demo SparkPost account we have created. This demo account has been verified to the sending domain "weetbicks.com" and we have setup a test email address "sparkpost@weetbicks.com" to act as the sender.
You may find that your emails will not send if you use the demo file with a different sender email address. If you wish to do this you should create your own SparkPost account, and obtain your own API key and verified sending domain to use in the example file.
Example File
We've included a Demo file to help you get started on your road to SparkPost integration. You can download using the link provided.
Click here to download the Sparkpost example file.
You can also check out our Mandrill example file for comparison below.
Click here to download the Mandrill example file.
Something to say? Post a comment...
Comments
Daniel Wood 31/07/2018 10:43am (6 years ago)
hi Eligio,
I'm not entirely sure to be honest, for our integrations the deprecation of TLSv1.0 had no impact, because as far as I'm aware FileMaker does not use TLS1.0, and things just continued to work exactly as they had. Are you saying that your integration between Sparkpost/FM has broken as a result of 1.0 deprecation?
Eligio Ospina Meneses 27/07/2018 1:59pm (6 years ago)
Dear Sir: I appreciate your help in solving the difficulty that has arisen with the depreciation of TLSv1.0. The indications of Sparkpost are not clear to me. Could you tell me how to solve the problem? Thank you.
Virge 13/06/2018 6:29pm (6 years ago)
Probably good to mention the need to clean your SparkPost mailing list to remove invalid emails. If you're using the SparkPost lists, you can just use MailboxValidator to import and clean your list. Cleaned list will be automatically updated back to SparkPost.
https://www.mailboxvalidator.com/resources/articles/how-to-import-email-list-from-sparkpost/
If you're using the SparkPost API for sending, you can easily integrate the free MailboxValidator API to validate the email address before sending.
https://www.mailboxvalidator.com/api-single-validation
Daniel Wood 06/10/2017 7:24am (7 years ago)
thanks!
Daniel 04/08/2017 3:38am (7 years ago)
Hi Daniel, et al,
It appears that it may be possible to use an "Insert from URL" script step for sending with SparkPost. It should be 100% possible with FileMaker 16, because (thankfully!) FM has added cURL options support! However, I am referring to FileMaker 15 and below.
According to SparkPost's API, you can use "basic auth with the API key as username and an empty password." My initial test is positive and I'm proceeding to make more in-depth tests.
My solutions have used BaseElements plugin on FMS, but because the plugin is not "thread safe" the FMS Script Engine often crashes with multiple calls using BE!
Luc Marivoet 01/08/2017 2:58am (7 years ago)
Hi,
In your example is it possible to use a template-id which is passed through a variable field.
I want to sent mails to a selection of clients using a template-id which I created in Sparkpost with some replacement variables.
I could replace variables in my template but found no way to use a template as field. Now it is fixed in the custom function.
Thanks for some enlightment
Luc
Jimmy Sun 02/05/2017 8:42am (8 years ago)
Do you know of any bugs between FMP and SparkPost where DKIM fails on the receiving end? SPF and DKIM had been set up in DNS and verified. Same FMP script send through Mailjet, Mailgun, SendGrid, SMTP.com, and Postmark just fine, but as soon as server info is changed to SparkPost DKIM fails. If I send a manual message through Outlook via SparkPost then DKIM passes. Any thoughts?
Daniel Wood 04/04/2017 8:22am (8 years ago)
hi Ruben and Luc, apologies for the link having been broken, I accidentally deleted the file off our server. It took a while to find a copy of it but managed to track one down, and it is back up for you to download now. cheers.
Ruben 03/04/2017 8:16pm (8 years ago)
Hi Daniel,
I am trying to setup a solution using Sparkpost and I would love to see your example, but as Luc pointed out, the link dose not work.
Can you send me that example file?
Thank you!
Ruben
Luc Marivoet 02/04/2017 2:58am (8 years ago)
Hi Daniel,
I'm trying to setup a solution for sending mail through filemaker. Initially I used your mandril setup, but as this is becoming a paid option from MailChimp i'm looking to Sparkpost.
I was trying to download your example file for Sparkpost but i'm getting a file not found.
Any reason this file is missing?
Could you forward this example?
Thanks
Luc
Antonio 22/10/2016 12:04am (8 years ago)
I need to have the possibility to choice (browsing my file) the image to put on the body of the email.
Antonio 21/10/2016 10:05pm (8 years ago)
hi Daniel,
can i insert the image in the body of the message? not as attachment.
Antonio
Daniel Wood 17/08/2016 3:45pm (8 years ago)
hi Micael,
Thanks for pointing this out. I recently cleared out some files in this folder and didn't realise that I was sharing the SparkPost file from this location - it should be back there now and the link working again, apologies for the inconvenience.
Micael Blume 16/08/2016 4:49pm (8 years ago)
Hi Daniel, good work, I have hade a lot of help from the Mandrill article. But the http://share.teamdf.com/daniel/SparkPost.zip dosen't seem to work?
/micael
Daniel Wood 12/05/2016 9:16am (9 years ago)
hi Tami, thanks for the question. Have a look in the article for the section "An important note on the API key and account used in the demo file".
in your own sparkpost setup have you verified your sending domain with SparkPost, either by way of adding SPF/DKIM records, or verifying a sent email to abuse/postmaster @ your domain ? In your account settings in SparkPost you'll find ways in which you can verify your domain. If you don't verify, then SparkPost will not send via your domain as it must ensure it is verified.
If you have, and are still seeing this issue, then your sending domain may be blacklisted by SparkPost. THere are certain domains that SparkPost will not send from for security and protection against spam reasons, such as gmail, hotmail and so on. These are not verified sending domains.
If you feel your domain is verified but is blacklisted, you should probably contact SparkPost support to enquire as to why it's on their blacklist.
cheers
Tami Williams 12/05/2016 12:29am (9 years ago)
I can send FileMaker SMTP email via SparkPost ONLY if do a regular FileMaker script with the 'send using SMTP server' option.
I can NOT send email using your demo, after putting in my SparkPost API key and using from & to email addresses with my domain name; I'm getting this error:
errors: message: Invalid domain
description: Unconfigured or unverified sending domain.
code: 7001
code: 7001 = Sending domain blacklisted.
This makes no sense to me b/c if it were true then sending using FileMaker 'send email' script step wouldn't work either.
Any ideas?
Is there some other SparkPost API key I should be using for the email method used in your demo?
Gouri 14/04/2016 10:31pm (9 years ago)
Hi ,
Earlier I used mandrill for sending bulk mail. Now I am migrating to sparkpost.I want implement this using c#.
I have read in few forums and informative links , sparkpost gives us two options to send request in JSON or XML.
Here comes my query, How do send request if choose XML.Is there any document for this.
Please do help me in this.
Thank you.
Eric 14/04/2016 3:50pm (9 years ago)
Oh .. nevermind .. I was also looking at the API docs for SendGrid, and it too requires an Authorization header. SendGrid wants the "Bearer" prefix.
Mea culpa, I got them confused.
Daniel Wood 14/04/2016 11:50am (9 years ago)
hi Eric, thanks for the comment. I'm curious as to why the word Bearer is needed?
Eric 14/04/2016 11:29am (9 years ago)
Great stuff, and very timely for me.
One erratum:
The Authorization header you need to send should look more like this:
BE_HTTP_Set_Custom_Header ( "Authorization" ; "Bearer " & <APIKEY> )
That is, the word "Bearer" followed by the APIKEY.
fulminic 05/04/2016 3:25am (9 years ago)
Sorry to spam your board. I promise i am done now :)
Here's the solution for anyone trying to send server-stored attachments. (This returns an additional line of data in the filename which results in an invalid JSON request.)
Edit the custom function sparkpost_attachmentCodePlugin and replace this line:
FileName = sparkpost_escapeData ( GetAsText ( thisAttachment ) ) ;
with this:
FileName = sparkpost_escapeData ( GetContainerAttribute ( thisAttachment;"filename" ) ) ;
Now the filename will only contain the actual attachment name and nothing else.
fulminic 05/04/2016 3:04am (9 years ago)
I have edited the custom function where i changed FileName to a fixed value (e.g. "test.doc") and then it works fine.
"\"name\": \"" & "test.doc" & "\"," &
The attachment is there, just not with it's own name but with the name test.doc.
So i need to figure out a way to remove this unnecessary extra line of data from the filename:
FILE:Secure/E3/79/28ECEXXX/90AXF5X31/CD64D69B/D8E0",
fulminic 05/04/2016 2:40am (9 years ago)
Unlike Mandrill, SparkPost seems to have problems dealing with secure server-stored attachments.
This line is in my JSON request:
{"content": {
"from": {
"name": "John Doe]",
"email": "john@doe.nl"},
"subject": "A Subject",
"html": "a text",
"attachments": [ {
"type": "application/octet-stream",
"name": "remote:form.doc\
FILE:Secure/E3/79/28ECE104/90AF5C31/CD64D69B/D8E0",
"data": " ######"} ]},
"recipients": [{"address": {"name": "Jane Doe","email": "jane@doe.nl"}}]}
the JSON validation returns the error line 11, which is where the attachment is defined:
Error: Parse error on line 11:
...stream", "name": "remote:Application
----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got 'undefined'
I can send the email with no problems when i exclude the attachment. Any ideas?
Tommy Andersen 04/04/2016 10:07pm (9 years ago)
Referring to the SparkPost example file. I have been trying to modify the sample file to be able to send the same message to three different addresses, but I have some problems with the syntax. (Both a single address with a BCC and a CC, or three single addresses would work.)
What would be the best way of accomplishing that?
Daniel Wood 30/03/2016 11:10am (9 years ago)
hi Yeison, sorry I am not sure how insert URL method can be used with SparkPost at this stage, hopefully someone can figure this out, I just don't have time to figure this out myself.
Hi Joan, in order to use SparkPost API from your sparkpost account you need to ensure you are sending from a valid sending domain. In the setup area for your account there is an area to verify your sending domain and there are a few ways in which to do this. Check the article section on this topic. Note domains like hotmail or gmail will not work.
1 2 next »
No one has commented on this page yet.
RSS feed for comments on this page | RSS feed for all comments