"our idea is that developers should know about details such as schemas and indexes (the Parse engineers strongly agreed in hindsight)."
Many engineers I have worked with like to throw around terms like: "CQRS", "Event sourcing", "no schema's", "document-based storage", "denormalize everything" and more. However, when pushed, I often see that they lack a basic understanding of DBMSes, and fill up this gap by basically running away from it. For 95% of the jobs, a simple, non-replicated (but backed-up) DBMS will do just fine. The remaining 3% might benefit from some sharding and the final 2% are for specialized databases, such as column-oriented DBMSes, document-oriented DBMSes or graph-based databases.
At this very moment, I'm working on a classical DBMS, searching through around 200 million tuples within the order of 2 seconds (partially stored columnar using psql array types and GIN indexes). Not for a second did I consider using any kind of specialized database, nor optimized algorithms, except for some application-level caching. Some problems are caused by DBMS to backend and backend-to-frontend network utilization, which is generally not something a specialized DBMS can solve.
You are assuming that 95% of the jobs will remain the same throughout their entire lifetimes. Which is not true and I'm sure many would prefer for a database not to become a liability in case they happen to attract a lot of users, which requires more availability than a classical DBMSes can provide.
The problem is more about constrains required to scale and achieve high-availabily and low-latency, rather than a database choice. But classical DBMSes lack those constrains and you may and probably will end up with a design that just cannot be scaled without removing and redesigning a lot of things that users rely on, and also with engineers that never learned how to do that, so you may end up in an urgent need of a distributed systems specialist too. Scary. You can avoid that by using systems designed for high-availability from the very beginning, like Riak. Same goes for blobs and filesystems, using POSIX API instead of an object storage is going to haunt you later on, because in your designs you may end up relying on things that ultimately cannot scale well and guaranty latency, as gitlab learned the hard way.
> Some problems are caused by DBMS to backend and backend-to-frontend network utilization, which is generally not something a specialized DBMS can solve.
I absolutely agree. RDBMSs are the kings of one-size-fits-most. In the NoSQL ecosystem, MongoDB starts growing into this role. Of course, many performance problems are not caused by the backend but rather the network (an average website makes 108 HTTP requests) or the frontend (Angular2 without AOT, I'm looking at you). We took an end-to-end approach to tackle this problem:
1. As objects, files and queries are fetched via a REST API, we can make them cachable at the HTTP level to transparently use browser caches, forward proxies, ISP interception proxies, CDNs and reverse proxies.
2. In order to guarantee cache coherence, we invalidate queries and objects in our CDNs and reverse proxy caches when the change using an Apache Storm-based real-time query matching pipeline.
3. To keep browser up-to-date we ship a Bloom filter to the client indicating any data that has a TTL that is non-expired but was recently written. This happens inside the SDK. Optionally client can subscribe to queries via websockets (as in Firebase or Meteor).
The above scheme is the core of Baqend and quite effective in optimizing end-to-end web performance. Coupled with HTTP/2 this gives us a performance edge of >10x compared to Firebase and others. The central lesson learned for us is that it's not enough to focus on optimizing one thing (e.g. database performance) when the actual user-perceived problems arise at a different level of the stack.
> At this very moment, I'm working on a classical DBMS, searching through around 200 million tuples within the order of 2 seconds (partially stored columnar using psql array types and GIN indexes).
hi, this comment was timely. I have been asking myself what to use to persist a 10 million node graph. There has been a lot of marketing around using Hbase (obscene to setup and maintain) and Cassandra (much better.. but paywalled).
My main db is postgresql and we use jsonb quite extensively. The problem is that there is a lot of statements that "a single huge jsonb table in postgresql will never scale. you will have made a horrible mistake".
Seems you have ample experience on this front - whats your take?
It seems like you're suggesting that "classical" DBMS are inherently relational. Pardon me if I've misread you but, respectfully, as awesome as relational databases are, you haven't offered any reason why they should be your default versus some kind of NoSQL store. Insofar as your concern is simplicity, which you say you prefer (understandably!), it's actually just as likely the simplest option will be non-relational. There's also nothing inherently specialized about a non-relational store; in particular, hierarchical and key-value stores have at least as much maturity in production as relational ones (cf. IMS and dbm, respectively).
(Sorry for the pedantry, but I both work on a cloud document-oriented database and recently watched a bunch of collegiate developers struggle to set up MySQL when their queries could easily have been handled by a simpler system.)
"searching through around 200 million tuples within the order of 2 seconds (partially stored columnar using psql array types and GIN indexes). Not for a second did I consider using any kind of specialized database, nor optimized algorithms, except for some application-level caching. "
--
That's cool and all, but for some real-time applications, retrieving in the order of seconds is unacceptable.
DBMS has their earned place in the backend stack but I found that statement of covering 98% of the cases a bit of a stretch.
As an Urban Airship employee from 2010-2013 this is eerily familiar stuff. I feel like we should all have a big "we tried and failed to use mongodb as a distributed oltp database" support group.
One point of disagreement:
> However, they had an unspoken value system of not trusting their users to deal with complex database and architectural problems.
At least for enterprise customers (who actually pay the bills) this is the correct value system. It's not that enterprise developers are bad. It's that enterprise development moves incredibly slowly, so the more freedom and options and implementation details you give them to work with the longer it will take them to be successful.
What you as a startup engineer may feel like is a powerful and expressive API just added 6-12 months to the development cycle of every enterprise customer... and thus many will give up on the integration.
Do you have a source for that? It is what the Parse engineers told me, but that kind of information is notoriously hard to corroborate. Any time I asked this question to MongoDB engineers they were evading concrete numbers for deployment sizes.
I would like to hear more about the introduction of LSM-based RocksDB storage into the Parse architecture. Specifically how long before they felt MMAP was no longer meeting their needs?
I was also curious about this?
>"Frequent (daily) master reelections on AWS EC2. Rollback files were discarded and let to data loss."
Was the velocity of rollbacks sufficiently high enough that they were't able to process them?
I thought Parse was a great company with a great product that served an important market. I'm still not sure why FB acquired them only to retire it.
We had issues with the MMAPv1 storage engine from the beginning, mostly due to it's lack of document level locking and storage bloat.
One of the biggest benefits from being acquired by Facebook was working with the RocksDB team to come up with MongoDB +RocksDB, but this was also happening around the time when Parse was starting to wind down.
Being the only company running this new storage engine freaked us the fuck out. So we tried to blog and give talks as much as possible to get other folks interested and willing to test it out.
Author of the post here. If you have additional key insights about the Parse infrastructure, please post them here and I will directly add them to the article.
We were a parse user for (many) apps, and tried to run parse server briefly before just letting all the features built on it die.
I think one of the major instabilities not mentioned the request fanout. On startup the parse ios client code could fan out to upwards of 10 requests, generally all individual pieces of installation data. Your global rate limit was 30 by default. Even barely used apps would get constantly throttled.
It was even worse running ParseServer, since on Mongodb those 10 writes per user end up as a sitting in a collection write queue blocking reads until everything is processed. It was easier to kill the entire feature set it was based on.
I know there were a ton of other issues but I've forced myself to forget them.
What was the throughput (per server) after the Go rewrite? I'd imagine an async rewrite would be a lot more efficent than the 15-30 req/s that the Rails version was getting.
How often (if ever) did you experience data loss due to the MongoDB write concern = 1?
I'm surprised vendor lock-in was not mentioned as a problem. I suppose it's a fundamental problem with offering "BaaS" and since the author is pushing his own BaaS offering, any critique of vendor lock-in may be unlikely.
I first used parse for a mobile app that grew to 600k users. I was totally against adopting parse, for a number of reasons including the unpredictable billing, failing background jobs, and the need for retry logic in every single write query. But my biggest issue was with vendor lock-in. When your app succeeds, and the backend is dependent on parse being in business, parse becomes a liability. Eventually you will need to take on a project to "migrate off parse." And you know what? That sucks for parse, because they were counting on power users to pay the big bills. But in reality, once a customer became a "power user," they started the process of migrating off parse.
When parse shutdown, I initially felt vindicated - ha! Vendor lock-in. But they handled the shut down and transition to open source extremely well. As a result, parse-server is now an open sourced, self hosted product with all the benefits of Parse.com but without the vendor lock-in!
I've been using parse-server exclusively for new projects. I'm very happy with it and it is always improving (though backwards compatibility has been hit or miss... but that's the price of rapid releases). It's very easy to setup, and does a lot of crud for you (user registration, email confirmation, login, forgot password, sessions). You can call the API from an open source SDK in almost any language on any platform (note: I'm the maintainer of ParsePy). Also, because you have direct access to mongo, you can optimize query performance outside of parse. For example you can create unique indexes, where with parse you had to query for duplicate objects at application level. There's even initial support for Postgres instead of mongo. Also, the dashboard is nice for non-technical people on your team to understand your data structures and ask more informed questions.
I'm not sure I would ever use another BaaS. It just seems like such a dumb decision to offload the entire backend of your app to a proprietary service. If the service was totally open source from the beginning, with a self hosted option, then I would consider it. At least that eliminates the threat of vendor lock-in. I get the feeling that the excellent handling of parse.com shutdown was an exception to the rule. I don't want to take unnecessary risks with the most important code of a project.
I agree, the parse shutdown was organized extremely well. The open source parse server, one year of migration time and a ton of new vendors that now offer to host your parse app, all made it much easier to handle the shutdown. It's also great to see the community still working on the open source server.
That said, there are a lot of upsides to having a company work full-time on your proprietary cloud solution and ensure its quality and availability. If an open source project dies or becomes poorly maintained you are in trouble too. Your team might not have the capacity to maintain this complex project on top of their actual tasks.
Also open sourcing your platform is a big risk for a company. Take RethinkDB for example: Great database, outstanding team but without a working business model and most recently without a team working full time, it is doomed to die eventually.
Nevertheless, we try to make migrating from and to Baqend as smooth as possible. You can import and export all your data and schemas, your custom business logic is written in Node.js and can be executed everywhere. You can also download a community server edition (single server setup) to host it by yourself.
Still a lot of users even require proprietary solutions and the maintenance and support that comes with it. And often they have good reasons, from requiring a maintenance free platform to to warranties or license issues. After all, a lot of people are happy to lock into AWS even though solutions based on OpenStack, Eucalyptus etc. are available.
"applications were often inconsistent" - I've heared this about parse before. Always thought this is due to using MongoDB. If you use the same database how can you enforce more consistency?
Although MongoDB has its limits regarding consistency, there are things that we do differently from parse to ensure consistency:
- The first thing is that we do not read from slaves. Replicas are only used for fault tolerance as it's the default in MongoDB. This means you always get the newest object version from the server.
- Our default update operation compares object versions and rejects writes if the object was updated concurrently. This ensures consistency for single object read-modify-write use cases. There is also an operation called "optimisticSave" the retries your updates until no concurrent modification comes in the way. This approach is called optimistic concurrency control. With forced updates, however, you can override whatever version is in the database, in this case, the last writer wins.
- We also expose MongoDBs partial update operators to our clients (https://docs.mongodb.com/manual/reference/operator/update/). With this, one can increase counters, push items into arrays, add elements into sets and let MongoDB handle concurrent updates. With these operations, we do not have to rely on optimistic retries.
- The last and most powerful tool we are currently working on is a mechanism for full ACID transactions on top of MongoDB. I've been working on this at Baqend for the last two years and also wrote my master thesis on it. It works roughly like this:
1. The client starts the transaction, reads objects from the server (or even from the cache using our Bloom filter strategy) and buffers all writes locally.
2. On transaction commit all read version and updated objects are sent to the server to be validated.
3. The server validates the transaction and ensures the isolation using optimistic concurrency control. In essence, if there were concurrent updates, the transaction is aborted.
4. Once the transaction is successfully validated, updates are persisted in MongoDB.
There is a lot more in the details to ensure isolation, recovery as well as scalability and also to make it work with our caching infrastructure. The implementation is currently in our testing stage. If you are interested in the technical details, this is my master thesis: https://vsis-www.informatik.uni-hamburg.de/getDoc.php/thesis...
would be intresting understing more about "Static pricing model measured in guaranteed requests per second did not work well." . what happened and what would have been a better solution afterwards.
The core problem was that the sustained number of requests did not really cause any bottlenecks. Actual problems were:
- At the Parse side: expensive database queries on the shared cluster that could not be optimized since developers had no control over indexing and sharding.
- At the customer side: any load peaks (e.g. a front page story on hacker news) caused the Parse API to run into rate limits and drop your traffic.
What is the reasoning behind not using sharding and homing each customer to a replica set?
How did you deal with customers growing large or spiking in usage? How did you manage such hot spots?
Sharding is one of the key benefits of MongoDB (and frankly most NoSQL solutions). Of course, you have to pick a good shard key.
One reason I can think of is that IMO sharding on MongoDB before version 2.4 had too many issues to be production reliable. If Parse started with MongoDB 2.2 or earlier, then I can see how they would avoid sharding.
There are definitely some similarities. Like in Gomix, the idea of Baqend is to let developers easily build websites, bots and APIs. However, Baqend has a stronger focus on how data is stored and queried, while Gomix is more focused on a rapid prototyping experience inside the browser.
In my opinion, the whole movement about BaaS is all about making things as smooth as possible for developers and shorten the time-to-market to its minimum. What some providers like Parse lost on the way, are the capabilities for small prototypes to grow to large systems. That requires being scalable at both the level of API servers, user-submitted backend code and the database system. And at some point, it also requires letting the user tune things like indices, shard keys and execution timeouts. This is the route we took at Baqend. We do not want to be the simplest solution out there, but we aim to provide the fastest (we use new web caching tricks) and most scalable one (we expose how the database is distributed).
If the database is the bottleneck, why are you using the same (MongoDB) database as Parse (and Firebase) were?
The problems described in the article are quite literally are my pitch deck, which I have successfully raised from billionaires Tim Draper and Marc Benioff of Salesforce, for http://gun.js.org/ . So why did you decide to stick with MongoDB when many other databases have tackled solving those problems?
The database was not the bottleneck for Parse, how they used it was. Actually that is the case for most databases systems, you have to choose a set of trade-offs your application can live with. We wrote about this in more detail here: https://medium.baqend.com/nosql-databases-a-survey-and-decis...
Word of advice, this post comes across as immature. What are you mentioning "billionaires" for in this context? And careful, with this audience the Theranos-enraged mob is a real liability
[+] [-] edejong|9 years ago|reply
Many engineers I have worked with like to throw around terms like: "CQRS", "Event sourcing", "no schema's", "document-based storage", "denormalize everything" and more. However, when pushed, I often see that they lack a basic understanding of DBMSes, and fill up this gap by basically running away from it. For 95% of the jobs, a simple, non-replicated (but backed-up) DBMS will do just fine. The remaining 3% might benefit from some sharding and the final 2% are for specialized databases, such as column-oriented DBMSes, document-oriented DBMSes or graph-based databases.
At this very moment, I'm working on a classical DBMS, searching through around 200 million tuples within the order of 2 seconds (partially stored columnar using psql array types and GIN indexes). Not for a second did I consider using any kind of specialized database, nor optimized algorithms, except for some application-level caching. Some problems are caused by DBMS to backend and backend-to-frontend network utilization, which is generally not something a specialized DBMS can solve.
[+] [-] zzzcpan|9 years ago|reply
The problem is more about constrains required to scale and achieve high-availabily and low-latency, rather than a database choice. But classical DBMSes lack those constrains and you may and probably will end up with a design that just cannot be scaled without removing and redesigning a lot of things that users rely on, and also with engineers that never learned how to do that, so you may end up in an urgent need of a distributed systems specialist too. Scary. You can avoid that by using systems designed for high-availability from the very beginning, like Riak. Same goes for blobs and filesystems, using POSIX API instead of an object storage is going to haunt you later on, because in your designs you may end up relying on things that ultimately cannot scale well and guaranty latency, as gitlab learned the hard way.
[+] [-] DivineTraube|9 years ago|reply
I absolutely agree. RDBMSs are the kings of one-size-fits-most. In the NoSQL ecosystem, MongoDB starts growing into this role. Of course, many performance problems are not caused by the backend but rather the network (an average website makes 108 HTTP requests) or the frontend (Angular2 without AOT, I'm looking at you). We took an end-to-end approach to tackle this problem:
1. As objects, files and queries are fetched via a REST API, we can make them cachable at the HTTP level to transparently use browser caches, forward proxies, ISP interception proxies, CDNs and reverse proxies.
2. In order to guarantee cache coherence, we invalidate queries and objects in our CDNs and reverse proxy caches when the change using an Apache Storm-based real-time query matching pipeline.
3. To keep browser up-to-date we ship a Bloom filter to the client indicating any data that has a TTL that is non-expired but was recently written. This happens inside the SDK. Optionally client can subscribe to queries via websockets (as in Firebase or Meteor).
The above scheme is the core of Baqend and quite effective in optimizing end-to-end web performance. Coupled with HTTP/2 this gives us a performance edge of >10x compared to Firebase and others. The central lesson learned for us is that it's not enough to focus on optimizing one thing (e.g. database performance) when the actual user-perceived problems arise at a different level of the stack.
[+] [-] sandGorgon|9 years ago|reply
hi, this comment was timely. I have been asking myself what to use to persist a 10 million node graph. There has been a lot of marketing around using Hbase (obscene to setup and maintain) and Cassandra (much better.. but paywalled).
My main db is postgresql and we use jsonb quite extensively. The problem is that there is a lot of statements that "a single huge jsonb table in postgresql will never scale. you will have made a horrible mistake".
Seems you have ample experience on this front - whats your take?
[+] [-] wsh91|9 years ago|reply
(Sorry for the pedantry, but I both work on a cloud document-oriented database and recently watched a bunch of collegiate developers struggle to set up MySQL when their queries could easily have been handled by a simpler system.)
[+] [-] nodamage|9 years ago|reply
Just curious, does this kind of performance require dedicated hardware or is it possible on platforms like AWS?
[+] [-] pvg|9 years ago|reply
[+] [-] pmelendez|9 years ago|reply
--
That's cool and all, but for some real-time applications, retrieving in the order of seconds is unacceptable.
DBMS has their earned place in the backend stack but I found that statement of covering 98% of the cases a bit of a stretch.
[+] [-] schmichael|9 years ago|reply
One point of disagreement:
> However, they had an unspoken value system of not trusting their users to deal with complex database and architectural problems.
At least for enterprise customers (who actually pay the bills) this is the correct value system. It's not that enterprise developers are bad. It's that enterprise development moves incredibly slowly, so the more freedom and options and implementation details you give them to work with the longer it will take them to be successful.
What you as a startup engineer may feel like is a powerful and expressive API just added 6-12 months to the development cycle of every enterprise customer... and thus many will give up on the integration.
[+] [-] tbrock|9 years ago|reply
They had the largest number of databases on a single replica set for sure, but largest in terms of cluster size or data size? No way.
[+] [-] DivineTraube|9 years ago|reply
[+] [-] bogomipz|9 years ago|reply
I was also curious about this? >"Frequent (daily) master reelections on AWS EC2. Rollback files were discarded and let to data loss."
Was the velocity of rollbacks sufficiently high enough that they were't able to process them?
I thought Parse was a great company with a great product that served an important market. I'm still not sure why FB acquired them only to retire it.
[+] [-] mkania|9 years ago|reply
One of the biggest benefits from being acquired by Facebook was working with the RocksDB team to come up with MongoDB +RocksDB, but this was also happening around the time when Parse was starting to wind down.
Being the only company running this new storage engine freaked us the fuck out. So we tried to blog and give talks as much as possible to get other folks interested and willing to test it out.
Parse now running MongoDB on RocksDB: http://blog.parse.com/announcements/mongodb-rocksdb-parse/
Strata: Open Source Library for Efficient MongoDB Backup: http://blog.parse.com/learn/engineering/strata-open-source-l...
MongoDB + RocksDB: Writing so Fast it Makes Your Head Spin: http://blog.parse.com/learn/engineering/mongodb-rocksdb-writ...
MongoDB + RocksDB: Benchmark Setup & Compression: http://blog.parse.com/learn/engineering/mongodb-rocksdb-benc...
[+] [-] DivineTraube|9 years ago|reply
[+] [-] 220|9 years ago|reply
I think one of the major instabilities not mentioned the request fanout. On startup the parse ios client code could fan out to upwards of 10 requests, generally all individual pieces of installation data. Your global rate limit was 30 by default. Even barely used apps would get constantly throttled.
It was even worse running ParseServer, since on Mongodb those 10 writes per user end up as a sitting in a collection write queue blocking reads until everything is processed. It was easier to kill the entire feature set it was based on.
I know there were a ton of other issues but I've forced myself to forget them.
[+] [-] koolba|9 years ago|reply
How often (if ever) did you experience data loss due to the MongoDB write concern = 1?
[+] [-] chatmasta|9 years ago|reply
I first used parse for a mobile app that grew to 600k users. I was totally against adopting parse, for a number of reasons including the unpredictable billing, failing background jobs, and the need for retry logic in every single write query. But my biggest issue was with vendor lock-in. When your app succeeds, and the backend is dependent on parse being in business, parse becomes a liability. Eventually you will need to take on a project to "migrate off parse." And you know what? That sucks for parse, because they were counting on power users to pay the big bills. But in reality, once a customer became a "power user," they started the process of migrating off parse.
When parse shutdown, I initially felt vindicated - ha! Vendor lock-in. But they handled the shut down and transition to open source extremely well. As a result, parse-server is now an open sourced, self hosted product with all the benefits of Parse.com but without the vendor lock-in!
I've been using parse-server exclusively for new projects. I'm very happy with it and it is always improving (though backwards compatibility has been hit or miss... but that's the price of rapid releases). It's very easy to setup, and does a lot of crud for you (user registration, email confirmation, login, forgot password, sessions). You can call the API from an open source SDK in almost any language on any platform (note: I'm the maintainer of ParsePy). Also, because you have direct access to mongo, you can optimize query performance outside of parse. For example you can create unique indexes, where with parse you had to query for duplicate objects at application level. There's even initial support for Postgres instead of mongo. Also, the dashboard is nice for non-technical people on your team to understand your data structures and ask more informed questions.
I'm not sure I would ever use another BaaS. It just seems like such a dumb decision to offload the entire backend of your app to a proprietary service. If the service was totally open source from the beginning, with a self hosted option, then I would consider it. At least that eliminates the threat of vendor lock-in. I get the feeling that the excellent handling of parse.com shutdown was an exception to the rule. I don't want to take unnecessary risks with the most important code of a project.
[+] [-] erikwitt|9 years ago|reply
That said, there are a lot of upsides to having a company work full-time on your proprietary cloud solution and ensure its quality and availability. If an open source project dies or becomes poorly maintained you are in trouble too. Your team might not have the capacity to maintain this complex project on top of their actual tasks.
Also open sourcing your platform is a big risk for a company. Take RethinkDB for example: Great database, outstanding team but without a working business model and most recently without a team working full time, it is doomed to die eventually.
Nevertheless, we try to make migrating from and to Baqend as smooth as possible. You can import and export all your data and schemas, your custom business logic is written in Node.js and can be executed everywhere. You can also download a community server edition (single server setup) to host it by yourself.
Still a lot of users even require proprietary solutions and the maintenance and support that comes with it. And often they have good reasons, from requiring a maintenance free platform to to warranties or license issues. After all, a lot of people are happy to lock into AWS even though solutions based on OpenStack, Eucalyptus etc. are available.
[+] [-] CurlyBraces|9 years ago|reply
[+] [-] erikwitt|9 years ago|reply
- The first thing is that we do not read from slaves. Replicas are only used for fault tolerance as it's the default in MongoDB. This means you always get the newest object version from the server.
- Our default update operation compares object versions and rejects writes if the object was updated concurrently. This ensures consistency for single object read-modify-write use cases. There is also an operation called "optimisticSave" the retries your updates until no concurrent modification comes in the way. This approach is called optimistic concurrency control. With forced updates, however, you can override whatever version is in the database, in this case, the last writer wins.
- We also expose MongoDBs partial update operators to our clients (https://docs.mongodb.com/manual/reference/operator/update/). With this, one can increase counters, push items into arrays, add elements into sets and let MongoDB handle concurrent updates. With these operations, we do not have to rely on optimistic retries.
- The last and most powerful tool we are currently working on is a mechanism for full ACID transactions on top of MongoDB. I've been working on this at Baqend for the last two years and also wrote my master thesis on it. It works roughly like this:
There is a lot more in the details to ensure isolation, recovery as well as scalability and also to make it work with our caching infrastructure. The implementation is currently in our testing stage. If you are interested in the technical details, this is my master thesis: https://vsis-www.informatik.uni-hamburg.de/getDoc.php/thesis...[+] [-] esseti|9 years ago|reply
[+] [-] DivineTraube|9 years ago|reply
- At the Parse side: expensive database queries on the shared cluster that could not be optimized since developers had no control over indexing and sharding.
- At the customer side: any load peaks (e.g. a front page story on hacker news) caused the Parse API to run into rate limits and drop your traffic.
[+] [-] astral303|9 years ago|reply
How did you deal with customers growing large or spiking in usage? How did you manage such hot spots?
Sharding is one of the key benefits of MongoDB (and frankly most NoSQL solutions). Of course, you have to pick a good shard key.
One reason I can think of is that IMO sharding on MongoDB before version 2.4 had too many issues to be production reliable. If Parse started with MongoDB 2.2 or earlier, then I can see how they would avoid sharding.
[+] [-] anilgulecha|9 years ago|reply
[+] [-] DivineTraube|9 years ago|reply
In my opinion, the whole movement about BaaS is all about making things as smooth as possible for developers and shorten the time-to-market to its minimum. What some providers like Parse lost on the way, are the capabilities for small prototypes to grow to large systems. That requires being scalable at both the level of API servers, user-submitted backend code and the database system. And at some point, it also requires letting the user tune things like indices, shard keys and execution timeouts. This is the route we took at Baqend. We do not want to be the simplest solution out there, but we aim to provide the fastest (we use new web caching tricks) and most scalable one (we expose how the database is distributed).
[+] [-] csmajorfive|9 years ago|reply
[+] [-] DivineTraube|9 years ago|reply
[+] [-] simscitizen|9 years ago|reply
[+] [-] DivineTraube|9 years ago|reply
[+] [-] marknadal|9 years ago|reply
The problems described in the article are quite literally are my pitch deck, which I have successfully raised from billionaires Tim Draper and Marc Benioff of Salesforce, for http://gun.js.org/ . So why did you decide to stick with MongoDB when many other databases have tackled solving those problems?
[+] [-] mdekkers|9 years ago|reply
gun.put({hello: "world"}).key('random/WVNqluP4q');
and
Databases can be scary
Who exactly are your target market?
[+] [-] DivineTraube|9 years ago|reply
[+] [-] redwood|9 years ago|reply