Monday, July 29, 2013

LEAN7: Migrate from MongoDB to AWS DynamoDB + SimpleDB

Migrate from MongoDB to DynamoDB + SimpleDB: New Server Side Architecture Ready for More Users

Recently we have 14,000 registered users, a small portion of them are paid users. I feel that TeamViz is recognized for more and more sales (even still a very small number) generated every month. However, I start to get trouble on our server architecture mentioned in this post. The issue is, the MongoDB backed database getting locked for unknown reason for several minutes every 2 hours. Initially, the all request will be hold for 2 minutes every 2 hours 7 minutes. Now it becomes more worse, all request will be hold for 7 minutes every 2 hours and 7 minutes. I asked this question on stackoverflow, but no answer yet. So I can either increase the capacity of servers, or shift to another database server. We are small, and I can try different solutions.

Because all the connection will be hold for several minutes, so the connection on load balancer looks like this way. (At the beginning I though the server are attacked, but no one will attack a sever every 2 hours 7 minutes, and for 1 month, right ^_^ )

So here are several possible solutions. Use another NoSQL database, or use managed NoSQL database. My first decision is to looking for other NoSQL database servers, I have read comparison of NoSQL solutionsthis link about NoSQL benchmark, and this link about couchbase. Every NoSQL database has some pros and cons.

I then talked with Kelly about the cost of server, cost of managed service, possibility of shifting to other NoSQL providers, or even shifting to MySQL. The conclusion is, current issue on MongoDB is just a start, we may take more time on managing databases and resolve performance, or some unknown issues. This will cost much energy. However, our focus is to providing better product. There are a lot of fun on playing NoSQL and other cutting edge technology. But that's not our goal. Shifting to managed database service can help us focusing on providing features/fix issues on product itself. At least we have a long list of features and issues to resolve. So we shifted to Amazon AWS DynamoDB, and to reduce the cost, part of the data on AWS SimpleDB. The server side is almost rewrote to handle the database change. I take this chance to practiced Promise pattern on node.js. It works great! and leveraged the middleware technology provided by Express framework. In addition, hold data of DynamoDB and SimpleDB in memcache. Everything has worked great for 24 hours (except that I got some error logs on memcache).

Here are the picture after 10 hours of migration. The huge periodically traffic disappeared.

Here are the new architecture on database and sync server.

You may have concern about accessing AWS from Linode, currently it's fine. We have more than 1.3 million items in one DynamoDB table, and response from DynamoDB to get one record by key is 25 ~ 45 ms from Linode network. SimpleDB has less than 20k items, and also 25 ~ 45 ms.

Some notes about the new architecture:
- Why Linode: much cheaper than AWS EC2.
- Why AWS DynamoDB and SimpleDB: don't want to worry about managing database.
- memcached suppose to work independently, we use CouchBase because they provided automatic clustering.
- Still, the design goal is to scale out. Every machine is independent. We can add more sync server and memcached server independently.
- Future plan: currently we still need a message queue, AWS SQS does not provide a way for post event to multiple subscribers simultaneously. RabbitMQ can make it. But message queue is not urgent so far.
- Future blog: I will share more experience on using SimpleDB and DynamoDB.


  1. You can use AWS SNS to post event to multiple subscribers, cannot you?
    - Future plan: currently we still need a message queue, AWS SQS does not provide a way for post event to multiple subscribers simultaneously. RabbitMQ can make it. But message queue is not urgent so far.

  2. This comment has been removed by the author.

  3. I understand you use CouchBase as a drop in replacement for memcached.

    Might I question why you didn't choose AWS Elasticache? I'm inclined to replicate what you do here, but we would use Elasticache instead. Comments?

  4. Great post! I look this up on the web and the first result on Google is you so i believe the search engine itself tells your trust-worthiness. I may want to have additional readings though but still great article about Teamviz.

  5. Really appreciate this wonderful post that you have provided for us.Great site and a great topic as well i really get amazed to read this. Its really good.texas ecommerce web development

  6. Very valuable post. Would love to hear your thoughts on FireBase from google. Outside of price consideration what would consider today?

    Thank you in advance for your time,

  7. Managed IT support plays an integral role in any business these days. Being that almost everything in the business world has gone digital, the ability to maintain the technological devices around us is a must.

  8. Thanks a lot very much for the high quality and results-oriented help.
    I won’t think twice to endorse your blog post to anybody who wants
    and needs support about this area.

    AWS Training in Chennai

    AWS Training in Bangalore

    AWS Training in Bangalore

  9. Excellent blog on how to develop your knowledge in Mongo db thanks for uploading the content. It is very much useful for those who search content related to technology.
    Salesforce Admin Training in Chennai
    Salesforce Administrator 201 Training in Chennai

  10. This really make a lot of sense and i have learned about the technology thanks for sharing Physiotherapy Clinic in OMR

  11. kudos For the great Blog With the Amazing Content Which Was Delivered Really Nice.Very much Useful for the beginners And the information Provided in this blog was genuine.
    With regards:
    Aws Certified Solutions Architect Training

  12. Amazon has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. Amazon Web Services (AWS) is a secure cloud services platform, offering compute power, database storage, content delivery and other functionality to help businesses scale and grow.For more information visit.
    aws online training
    aws training in hyderabad
    aws online training in hyderabad

  13. We have good game slots best casino games in history Get winnings here and now.

  14. This comment has been removed by the author.

  15. На любой вкус светодиодные ленты можно найти в Ekodio, бюджетные и премиум, всех цветов и характеристик

  16. I didn’t have enough good honest and conscientious people in my life so I was so scared to play all sorts of online games, slot machines, gamblingg But thank God, I got exactly on this site super gambling games real money ere, everything is honest and the conclusions are quick and everything is beautiful and clear.

  17. Very good information. we need learn with real time examples and for this we choose good training institute, who were interested to know about AWS which is quite interesting. We need a good training institute for my learning .. so people making use of the free demo classes.
    Many training institute provides free demo classes. One of the best training institute in Bangalore is Apponix Technologies.

  18. Thank you for the informative post about Security challenges in AWS , Found it useful . cloud migration services have now become secured and with no-risk

    cloud migration services

    aws cloud migration services

  19. I am definitely enjoying your website. You definitely have some great insight and great stories.
    VPS Hosting India

  20. Harus anda pahami dengan baik bahwa ada banyak alasan tertentu yang kemudian mengharuskan siapa saja bisa bermain
    bandar ceme
    freebet tanpa deposit
    paito warna terlengkap
    syair sgp

  21. No big surprise that pretty much every real tech organization as of now has a functioning AI program set up.machine learning certification

  22. This is the first & best article to make me satisfied by presenting good content.
    patio screen repair collier county

  23. Your article has proved your hard work and experience you have got in this field.
    tree service near me escondido

  24. Thanks a lot of you for giving us information by your awesome blog.

  25. great article! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    crypto bounties

  26. This post is very different and unique information. It's a very nice post and very attractive to me. Keep posting.

  27. I must appreciate you for providing such a valuable content for us. This is one amazing piece of article. Helped a lot in increasing my training in bangalore