top | item 18996200

JMAP: Like IMAP but Not Really

325 points| jasonmunro | 7 years ago |unencumberedbyfacts.com

228 comments

order

bad_user|7 years ago

FastMail is behind this protocol and from what I've read JMAP has evolved out of their web interface.

I've been a happy customer, even though lately I flirted with going back to GSuite for my personal email, but after a trial realized that Gmail does many things well, except for being a good email service. So I went back to FastMail and renewed for another 2 years.

Seeing this new protocol is exciting, because JMAP is being standardized at IETF. A breath of fresh air to see a new standard being developed.

Also from what I understand JMAP should be friendly for mobile usage. They kept notifications out of it, you're supposed to implement notifications using whatever the mobile platform provides. Interacting via JMAP is via plain HTTP requests, which is super cool.

I can totally see myself implementing a simple email client for automating online services. For example if you implement a commenting system for a website, you might want to do replies by email. That would be a cool project for me to try out.

I wonder if FastMail exposes JMAP publicly yet. Haven't seen any mentions in their admin or docs thus far.

eggsampler|7 years ago

I've been a happy Fastmail customer too, until I was made aware that you can impersonate other Fastmail customers by just spoofing the email address. Their servers just happily accept it. SPF and DKIM all pass with flying colours, and the only way you'd know it's happened is if you have DMARC on and happen to notice a pass in the report you don't remember sending. Well, that is if the recipient doesn't reply to the spoofed message - hope the damage wasn't already done though. It's effectively impossible for the recipient to know it's been spoofed.

The worst part is I think Fastmail is aware of it and just don't care (believe that's why they mark their emails with a green tick and text). I understand that email has never been really authenticated, but this just throws any trust I had in Fastmail out the window.

I will be evaluating other mail hosts at the end of my subscription.

josteink|7 years ago

> after a trial realized that Gmail does many things well, except for being a good email service.

This.

I remember back when Gmail was new and hot. It was unlike any other email service out there, and ridiculing people for using inferior email-solutions could to a certain extent be justified.

While other webmails were slow, had constantly reloading pages and what not, Gmail was fast. It was amazingly fast. Gone where the 5 minutes making the webmail work for you. You just sent the email and you were done. Just like that. Back then, this was unheard of.

These days though? Everyone is still pretending like GMail is the only game in town, when I think they have one of the worst webmail-interfaces out there. And it's slow. Oh god it is slow. And god forbid you try to load it in a browser not Chrome, because then it just grinds to a complete halt.

So yeah. Happy Firefox-using FastMail-customer here. You couldn't get me back to GMail even if you paid me.

inanutshellus|7 years ago

I considered FM a while back but... you pay per account! $50/account no less! I separate my email into three accounts across two domains and my wife does the same and I have kids who'll have their own email addresses at some point.

I can't even begin to consider using it at the price point I presume I'd be in.

Semaphor|7 years ago

> Also from what I understand JMAP should be friendly for mobile usage.

I read the FM blogs and I remember them mentioning that low battery consumption was one of the design goals.

The JMAP site [0] has:

> JMAP is designed to make efficient use of limited network resources. Multiple API calls may be batched in a single request to the server, reducing round trips and improving battery life on mobile devices.

[0]: https://jmap.io/spec-core.html

catdog|7 years ago

> Also from what I understand JMAP should be friendly for mobile usage. They kept notifications out of it, you're supposed to implement notifications using whatever the mobile platform provides. Interacting via JMAP is via plain HTTP requests, which is super cool.

Sounds actually not cool. Is there really no standardized way to do notifications? The world consists not only of shitty mobile walled gardens which insist on or strongly favor their proprietary implementations.

abtinf|7 years ago

> ...Gmail does many things well, except for being a good email service.

Gmail/Gsuite seem to be good email services from my perspective as lay user and occasional admin. Can you expand on why you think they are not good email services?

pedrocr|7 years ago

>JMAP is a REST API so it uses HTTP requests and responses to issue commands and get the results. Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”.

Describing this as REST is really strange. Defining your own operations over an HTTP POST is what SOAP and other RPC style web services do and specifically what REST isn't. But I guess that a lack of a standard behind REST ended up with the term being used for everything.

rswail|7 years ago

REST is what Roy Fielding wrote about in his thesis. It is an architectural style, not a protocol.

It involves two endpoints exchanging the state of a shared resource. It needs to be compliant with the constraints of that style.

People think that REST must be over HTTP, but it can be over any protocol. The essence is that it is a style of systems design, so JMAP can be considered RESTful as described in the link above.

REST is one of the real patterns in software architecture, a set of constraints, not a set of structural elements.

bradleyjg|7 years ago

At this point I guess any API that runs on top of HTTP is REST? Strange world.

mariusmg|7 years ago

Maybe unpopular opinion, but i'd take this any day over a "proper" REST api requiring DELETE and PUT.

Also i'd classify a REST api as anything that does http requests and consumes proper JSON. As long as this is true , the rest is squabbling :)

hardwaresofton|7 years ago

I really want to see some innovation in the email space. The landscape is like a sea of false promises and dashed dreams. SMTP is one of the bread-and-butters of the internet, yet it just doesn't seem to be moving forward (maybe that's for the best), and no one's building extensions on top of it.

Maybe I'm naive in thinking it was possible but we could have avoided this whole "make an account on X messenger so we can talk", if someone had just jammed XMPP and SMTP together.

Every few years I'm tempted to try and write a SMTP server (MX) myself but then realize that postfix has been around so long and is the go to choice for a reason. I don't have 20 years of figuring out how X mail server interprets SMTP to be as reliable as postfix. I settled for trying to wrap it[0].

[0]: https://gitlab.com/postmgr/postmgr

raimue|7 years ago

Yes, postfix has been around for so long and it is very solid. However, to run a mail server nowadays, it still needs a lot of configuration work and even additional daemons to add SPF checks, forwarding with SRS, and DKIM signatures. All of this is possible to be added though, which speaks for the flexibility of the postfix design.

PaulHoule|7 years ago

Email is "good enough".

The real hole is "instant messaging". Somehow every few years we got from ICQ to AIM to Paltalk to Skype to Facebook Messenger to Slack to ... (at least Altassian had the decency to take HipChat behind the woodshed and shoot it)

The strange thing is that these services dry up, blow away, get replaced, but they don't seem to improve on what came before.

There is a standard, XMPP, but the only people who care about it are firefighters, cops, and soldiers. For all the anger at internet giants these days, I can't see for the life of me why people aren't pushing for open instant messenging.

e12e|7 years ago

> if someone had just jammed XMPP and SMTP together.

Xmpp is/supports federated/ion (user1@server1.com can message user2@server2.com).

But the major players (by numbers) wanted silos: Google (talk) and fasebook (first gen of messenger).

They basically went lol, screw users (arguably because: spam. But hello, Gmail? And today fb spam...).

So thank Google and Facebook for deliberately gimping it so you can't fb message user@gmail.com or gtalk first name.lastname123@facebook.com.

amanzi|7 years ago

"if someone had just jammed XMPP and SMTP together" - wasn't that kind of what Google Wave was aiming for?

pjc50|7 years ago

It was sometimes possible to hold realtime conversations over email, before the amount of required spam processing became too high. There's no protocol reason why you can't do that between servers, although getting it into a client requires either IMAP push or something more modern.

> make an account

You are required to make an account somewhere so your account can be banned if you spam, basically.

wwarner|7 years ago

Apache James is a great smtp and imap server that has a lot of really great features, like distributed storage in cassandra and some support for jmap.

elFarto|7 years ago

Honestly I'm surprised no one has suggested replacing SMTP with a HTTP POST request. All you would need is an email address -> HTTP URL conversion (which I believe something like WebFinger already has). Then just POST your message to that URL and you're done. ISPs can run proxies that just authenticate that requests are from users and forwards them on.

pjmlp|7 years ago

Emails are perfectly fine as they are. We also don't keep re-inventing forks, umbrellas, hammers and such.

ggm|7 years ago

The main failure in mail standards is a lack of explicit utf8 clean support lhs@rhs -if this got fixed (it's often called universal acceptance) a lot of things about mail as an ecology would improve.

I have view on the spam thing. The whole "your idea will not work because" meme is hugely destructive of innovation in email. It sucks energy and mindshare. It's classic old timer put down.

What would (imnsho opinion) have fixed spam is sender pays. I've debated this with a lot of people. We're 50/50 on it. Fifty agree with me, fifty million don't.

Jmap is utf8 clean btw.

(I used to do email for a living in the eighties when life was simple and bang!chains!worked)

majewsky|7 years ago

> What would (imnsho opinion) have fixed spam is sender pays. I've debated this with a lot of people. We're 50/50 on it. Fifty agree with me, fifty million don't.

I propose a scheme where the sender pays e.g. 5 ct per e-mail (low enough that it does not matter for legitimate use, but high enough to make spam unprofitable). BUT with the following twist: The receiver can generate API tokens that allow free e-mail delivery.

So when I sign up for e.g. Twitter, I can give them an API token for my mail address so they can send notification mails to me for free. If they decide to start spamming me, or if they decide to sell my token to a spammer, I can just revoke the token and the spam flood stops instantly.

I have not found any flaws in this idea yet. RFC!

efdee|7 years ago

If paying to send email is your suggested solution, I can see why you don't like the meme in question. But the suggestion is rife with problems. How exactly are you going to get everyone to start paying for email, which is free today? If it's a parallel system, how are you going to get people to switch? Who are they going to pay?

oarsinsync|7 years ago

> What would (imnsho opinion) have fixed spam is sender pays. I've debated this with a lot of people. We're 50/50 on it. Fifty agree with me, fifty million don't.

Along with the billions of spam SMS that are also supposed to be sender-pays, sadly

zimpenfish|7 years ago

> What would (imnsho opinion) have fixed spam is sender pays.

Alternate but similar - sender holds the email until the receiver fetches: https://cr.yp.to/im2000.html

I'm not sure it actually helps that much to cut down spam but anything is worth a try at this point.

m_mueller|7 years ago

How about a combination of sender-pays and a whitelist for everyone else? Essentially, have a web-standard for what linkedin is doing.

jeltz|7 years ago

Paying might reduce the spam, but some spammers are perfectly fine with paying. Just look at text spam and robocallers.

c22|7 years ago

Sender pays to stuff junk in my physical mailbox and I have a lot more trouble with spam there than I do in my email.

arendtio|7 years ago

To give some context on why JMAP is relevant today, you might want to read this answer in the conversations FAQ [1]. It basically states that XMPP powered Apps are having trouble fighting against the battery saving software because XMPP is stateful. JMAP, on the other hand, is stateless.

As protocols are meant to connect different implementations and bringing more protocols for the same task quickly hurts the interoperability (before bringing some improvements in the long-term), I am skeptical to the effects those new protocols bring to the ecosystem. I am aware that JMAP was born more out of the RESTful requirements of an HTTP based App, but in general, I am wondering where this road will lead us.

[1]: https://github.com/siacs/Conversations#but-why-do-i-need-a-p...

Leace|7 years ago

Note that "RESTful" in JMAP's case means "everything over JSON HTTP POST", methods to be invoked are embedded in the JSON request.

As for XMPP battery consumption I didn't see major problems, Conversations.im is always <1% (I just checked and it shows 0%). On the other hand Conversations can use push to optimize battery usage.

billpg|7 years ago

Last time I looked at IMAP, I was a little peeved that servers didn't accept messages added to the "Outbox" folder and send them, instead requiring that clients talk SMTP and then (or not) save a copy of the same message to the "Sent" folder.

Could JMAP replace SMTP as well as IMAP?

raimue|7 years ago

Storing sent mails is really one of the oldest problems of the SMTP/IMAP combination.

BURL [1] is an SMTP extension that allows to submit a mail that is already stored on an IMAP server. This still requires to make a separate connection to send the mail, but at least it does not require uploading it to both SMTP and IMAP only to store it in the Sent folder.

However, this extension is rarely implemented or deployed, although the RFC itself is already twelve years old. As an example, Dovecot started to support it by acting as a SMTP proxy [2] as of version 2.3 in 2017. I am not aware of any mail clients that support BURL.

[1] https://tools.ietf.org/html/rfc4468

[2] https://wiki.dovecot.org/Submission

jasonmunro|7 years ago

JMAP actually does support replacing both. My next TODO item for Cypht is integrating SMTP support with JMAP.

akvadrako|7 years ago

Why do you think you need a new protocol for this? Instead what you want is a different IMAP server.

akie|7 years ago

The article mentions that JMAP does calendars as well. Anyone have a quick status update on how it compares to CalDav and if it's supported by any of the major calendar clients or servers? I did a quick google and looked at the JMAP site but couldn't find any info.

Background for this is that I implemented a calendar client and a calendar server (with SabreDAV) last year and I'm wondering if I should be adding support for JMAP anytime soon.

nmjenkins|7 years ago

Editor of the JMAP specs here. The JMAP calendar spec is currently just a draft, but now JMAP core and mail are (more or less) finalised and JSCalendar (https://tools.ietf.org/html/draft-ietf-calext-jscalendar-11) is in last call too, we will be looking to take the JMAP Calendar spec through the IETF process in the very near future. Since it is essentially just combing core with the JSCalendar data format, this should be reasonably straight forward.

sigsergv|7 years ago

Current state of calendaring is horrible, it's basically not possible right now to create compatible and reliable library/application.

aorth|7 years ago

Post is about JMAP (Fastmail's IMAP/SMTP replacement), by the author of a newish webmail client called Cypht that has just recently implemented support for JMAP. Cool! Keep going!

https://cypht.org/

bobm_kite9|7 years ago

Loved this bit:

> Internet: Do you think JMAP will really take off?

> Me: JMAP is an open, smart, modern, and powerful E-mail protocol, so probably not.

mverwijs|7 years ago

I've started to migrate 18 years of email history to Fastmail, after running my own mail infra for almost 2 decades.

I am not impressed. Folders moved in the webinterface show up unmoved in Thunderbird. And vice versa.

Sometimes it works, sometimes it doesn't. They're investigating the problem at the moment, but the updates they've given me don't give me much hope.

I'm looking at migrating away to another provider that just does plain old imap.

mschuster91|7 years ago

From experience, Exchange's Outlook API ("EWS") is pretty decent. It's XML-SOAP, sure, but there are libraries for it that can be readily used. The only thing they did fuck up is the three different kinds of IDs for an object (esp. confusing when accessing delegated team mailboxes/calendar events) but once you get it how it works, it's straightforward and allows you access to anything from email over calendar to address-book.

I just do not know if EWS is an "open standard" that could be replicated by a third party server or if it is only "open documentation" and licensed for Exchange only :(

amaccuish|7 years ago

I've sadly been looking for, and never found, an open source EWS <-> IMAP/(Cal/Card Dav) gateway. I'd love it if my users could get a full experience using Outlook for Mac for example. Right now I recommend them to just use Apple Mail/Calendar/Contacts or Thunderbird with the tb-sync extensions.

zAy0LfpBZLC8mAC|7 years ago

> JMAP is not designed around a persistent network socket, so it’s perfect for webmail clients that connect, do stuff, then disconnect (which is exactly NOT how IMAP is supposed to be used)

WTF? So, webmail was limited by the fact that it was running in a browser and thus had to use HTTP, which has semantics that don't really fit the needs of an email access protocol. Now, we do have stuff like websockets that would make it possible to run a protocol from the webmail client that actually fits the needs, and instead people invent a new protocol that inherently doesn't match the needs of the application?

majewsky|7 years ago

Most mail clients are running on mobile (judging by number of devices). Long-running connections on mobile are a bad idea because of network reliability and battery concerns.

andridk|7 years ago

I was really hoping for a GraphQL API standard for mail before reading this. But this sounds good too.

chrismorgan|7 years ago

Understand that JMAP is an object synchronisation protocol. It’s most interested in the problems that are out of scope for GraphQL.

You could have a variant of JMAP with GraphQL syntax and semantics, but there would be a fair bit of mismatch, for their purposes and focuses are quite different: JMAP is concerned with object identity and synchronisation, for what you might call thick clients (and JMAP Mail needs to be broadly IMAP compatible), while GraphQL is UI-focused, for what might reasonably be considered to be thin clients (not that they are logicless, but that they are very much focused on offloading burden to the server). I see no compelling case for a JMAP-like GraphQL thing.

It could still be an interesting task to develop a object synchronisation protocol like the JMAP core on top of GraphQL.

thomasfoster96|7 years ago

Reading through the JMAP website [0], I'm actually wondering whether whoever's behind JMAP has looked at GraphQL. There seem to be a lot of similarities (especially the "why not REST" section).

[0] https://jmap.io/index.html

nothrabannosir|7 years ago

Very interesting:

> JMAP is not designed around a persistent network socket, so it’s perfect for webmail clients that connect, do stuff, then disconnect

This would allow implementing JMAP in serverless architecture, e.g. self hosting on AWS lambda. That should be very cheap and significantly lower the barrier to self hosting (not needing to manage a box, just providing aws credentials).

Think of what it takes to truly decentralise email. There are technical hurdles, but also some fundamental ones:

- other providers mark you as spam because you’re unknown

- you need an always on server to actually get your incoming mail

This would at least solve the second problem. If we can develop a product like mailinabox (Which has its flaws but it’s the right idea), but instead of asking for a fresh vm, it just asks for aws credentials, that could be pretty solid.

Hopefully one day we can give the people back their control over their means of communication. This seems like a step in the right direction!

elcomet|7 years ago

I'm not sure I understand: JMAP is designed for communication between the client and the mail server. But you still would need a mail server, always on, to receive the incoming mail from other servers, right ?

Did I miss something about JMAP?

tjoff|7 years ago

Why?

Mail is decentralized already. Part from relying on DNS.

And you seem very eager to put everything in the amazon basket.

jannes|7 years ago

If everyone hosts their email with Amazon it's gonna be quite centralised.

pjc50|7 years ago

The first problem is actually the more serious one, sadly.

shittyadmin|7 years ago

So, you say that it doesn't maintain a persistent connection, but that poses a problem - it needs a way to do push notifications still I assume? Those need an open socket somewhere - does JMAP allow for it or does it make you rely on a 3rd party?

Mailtemi|7 years ago

If you are mention Ios/Android, yes you need the proxy to redirect push calls from JMAP server to the push services of apple/google.

youdontknowtho|7 years ago

IMAP is just a way for attackers to perform credential spray's against large providers these days.

busfahrer|7 years ago

I recently thought about email being based on such an old standard and then I wondered if maybe it didn't matter anymore, because most of the web is moving to gmail, and my guess is (and somebody please correct me) that when I send an email from gmail to gmail, that no actual SMTP is involved? I'm assuming that they have some internal protocol that is just adapter'd to "Email" at the very end?

pjc50|7 years ago

> most of the web is moving to gmail

This isn't true; there's a LOT of industry on Exchange of some sort, and plenty of older institutions running their own email. Especially in IETF land.

And it's a huge risk increase if everyone moves everything to gmail.

beagle3|7 years ago

It fakes it, at the very least - you can do a “show original message” and see all the RFC.822 and mine headers.

Microsoft Exchange used to not bother with actually SMTPing when talking to another exchange server (or itself), I don’t know what it does these days.

mattnewport|7 years ago

I hear of a lot more people moving away from Gmail than moving to Gmail.