Monday, July 29, 2013

LEAN7: Migrate from MongoDB to AWS DynamoDB + SimpleDB

Migrate from MongoDB to DynamoDB + SimpleDB: New Server Side Architecture Ready for More Users

Recently we have 14,000 registered users, a small portion of them are paid users. I feel that TeamViz is recognized for more and more sales (even still a very small number) generated every month. However, I start to get trouble on our server architecture mentioned in this post. The issue is, the MongoDB backed database getting locked for unknown reason for several minutes every 2 hours. Initially, the all request will be hold for 2 minutes every 2 hours 7 minutes. Now it becomes more worse, all request will be hold for 7 minutes every 2 hours and 7 minutes. I asked this question on stackoverflow, but no answer yet. So I can either increase the capacity of servers, or shift to another database server. We are small, and I can try different solutions.

Because all the connection will be hold for several minutes, so the connection on load balancer looks like this way. (At the beginning I though the server are attacked, but no one will attack a sever every 2 hours 7 minutes, and for 1 month, right ^_^ )


So here are several possible solutions. Use another NoSQL database, or use managed NoSQL database. My first decision is to looking for other NoSQL database servers, I have read comparison of NoSQL solutionsthis link about NoSQL benchmark, and this link about couchbase. Every NoSQL database has some pros and cons.

I then talked with Kelly about the cost of server, cost of managed service, possibility of shifting to other NoSQL providers, or even shifting to MySQL. The conclusion is, current issue on MongoDB is just a start, we may take more time on managing databases and resolve performance, or some unknown issues. This will cost much energy. However, our focus is to providing better product. There are a lot of fun on playing NoSQL and other cutting edge technology. But that's not our goal. Shifting to managed database service can help us focusing on providing features/fix issues on product itself. At least we have a long list of features and issues to resolve. So we shifted to Amazon AWS DynamoDB, and to reduce the cost, part of the data on AWS SimpleDB. The server side is almost rewrote to handle the database change. I take this chance to practiced Promise pattern on node.js. It works great! and leveraged the middleware technology provided by Express framework. In addition, hold data of DynamoDB and SimpleDB in memcache. Everything has worked great for 24 hours (except that I got some error logs on memcache).

Here are the picture after 10 hours of migration. The huge periodically traffic disappeared.

Here are the new architecture on database and sync server.

You may have concern about accessing AWS from Linode, currently it's fine. We have more than 1.3 million items in one DynamoDB table, and response from DynamoDB to get one record by key is 25 ~ 45 ms from Linode network. SimpleDB has less than 20k items, and also 25 ~ 45 ms.

Some notes about the new architecture:
- Why Linode: much cheaper than AWS EC2.
- Why AWS DynamoDB and SimpleDB: don't want to worry about managing database.
- memcached suppose to work independently, we use CouchBase because they provided automatic clustering.
- Still, the design goal is to scale out. Every machine is independent. We can add more sync server and memcached server independently.
- Future plan: currently we still need a message queue, AWS SQS does not provide a way for post event to multiple subscribers simultaneously. RabbitMQ can make it. But message queue is not urgent so far.
- Future blog: I will share more experience on using SimpleDB and DynamoDB.

126 comments:

  1. You can use AWS SNS to post event to multiple subscribers, cannot you?
    Quote:
    - Future plan: currently we still need a message queue, AWS SQS does not provide a way for post event to multiple subscribers simultaneously. RabbitMQ can make it. But message queue is not urgent so far.

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. I understand you use CouchBase as a drop in replacement for memcached.

    Might I question why you didn't choose AWS Elasticache? I'm inclined to replicate what you do here, but we would use Elasticache instead. Comments?

    ReplyDelete
  4. Great post! I look this up on the web and the first result on Google is you so i believe the search engine itself tells your trust-worthiness. I may want to have additional readings though but still great article about Teamviz.

    ReplyDelete
  5. Really appreciate this wonderful post that you have provided for us.Great site and a great topic as well i really get amazed to read this. Its really good.texas ecommerce web development

    ReplyDelete
  6. Very valuable post. Would love to hear your thoughts on FireBase from google. Outside of price consideration what would consider today?

    Thank you in advance for your time,
    Emmanuel

    ReplyDelete
  7. Managed IT support plays an integral role in any business these days. Being that almost everything in the business world has gone digital, the ability to maintain the technological devices around us is a must.

    ReplyDelete
  8. Thanks a lot very much for the high quality and results-oriented help.
    I won’t think twice to endorse your blog post to anybody who wants
    and needs support about this area.


    AWS Training in Chennai


    AWS Training in Bangalore


    AWS Training in Bangalore

    ReplyDelete
  9. Excellent blog on how to develop your knowledge in Mongo db thanks for uploading the content. It is very much useful for those who search content related to technology.
    Regards:
    Salesforce Admin Training in Chennai
    Salesforce Administrator 201 Training in Chennai

    ReplyDelete
  10. This really make a lot of sense and i have learned about the technology thanks for sharing Physiotherapy Clinic in OMR

    ReplyDelete
  11. kudos For the great Blog With the Amazing Content Which Was Delivered Really Nice.Very much Useful for the beginners And the information Provided in this blog was genuine.
    With regards:
    Aws Certified Solutions Architect Training

    ReplyDelete
  12. Amazon has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. Amazon Web Services (AWS) is a secure cloud services platform, offering compute power, database storage, content delivery and other functionality to help businesses scale and grow.For more information visit.
    aws online training
    aws training in hyderabad
    aws online training in hyderabad

    ReplyDelete
  13. We have good game slots best casino games in history Get winnings here and now.

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. На любой вкус светодиодные ленты можно найти в Ekodio, бюджетные и премиум, всех цветов и характеристик

    ReplyDelete
  16. I didn’t have enough good honest and conscientious people in my life so I was so scared to play all sorts of online games, slot machines, gamblingg But thank God, I got exactly on this site super gambling games real money ere, everything is honest and the conclusions are quick and everything is beautiful and clear.

    ReplyDelete
  17. Very good information. we need learn with real time examples and for this we choose good training institute, who were interested to know about AWS which is quite interesting. We need a good training institute for my learning .. so people making use of the free demo classes.
    Many training institute provides free demo classes. One of the best training institute in Bangalore is Apponix Technologies.
    https://www.apponix.com/AWS-Essentials-Solutions-Architect/AWS-solution-arcchitect-training-in-bangalore.html

    ReplyDelete
  18. Thank you for the informative post about Security challenges in AWS , Found it useful . cloud migration services have now become secured and with no-risk

    cloud migration services

    aws cloud migration services

    ReplyDelete
  19. I am definitely enjoying your website. You definitely have some great insight and great stories.
    VPS Hosting India

    ReplyDelete
  20. Harus anda pahami dengan baik bahwa ada banyak alasan tertentu yang kemudian mengharuskan siapa saja bisa bermain
    asikqq
    http://dewaqqq.club/
    http://sumoqq.today/
    interqq
    pionpoker
    bandar ceme
    freebet tanpa deposit
    paito warna terlengkap
    syair sgp

    ReplyDelete
  21. No big surprise that pretty much every real tech organization as of now has a functioning AI program set up.machine learning certification

    ReplyDelete
  22. This is the first & best article to make me satisfied by presenting good content.
    patio screen repair collier county

    ReplyDelete
  23. Your article has proved your hard work and experience you have got in this field.
    tree service near me escondido

    ReplyDelete
  24. Thanks a lot of you for giving us information by your awesome blog.
    123movies

    ReplyDelete
  25. great article! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    crypto bounties

    ReplyDelete
  26. This post is very different and unique information. It's a very nice post and very attractive to me. Keep posting.
    finecoin

    ReplyDelete
  27. I must appreciate you for providing such a valuable content for us. This is one amazing piece of article. Helped a lot in increasing my knowledge.aws training in bangalore

    ReplyDelete
  28. Great article and a nice way to promote online. I’m satisfied with the information that you provided tree trimmers palm beach county

    ReplyDelete
  29. I must appreciate you for providing such a valuable content for us. This is one amazing piece of article. Helped a lot in increasing my knowledge.aws training in bangalore

    ReplyDelete
  30. Really a awesome blog for the freshers. Thanks for posting the information.devops training in bangalore


    ReplyDelete
  31. Great Article. Thank you for sharing! Really an awesome post for every one.

    IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

    JavaScript Training in Chennai

    JavaScript Training in Chennai


    ReplyDelete
  32. Very nice bro, thanks for sharing this with us. Keep up the good work and Thank you for sharing information
    tree removal west palm beach

    ReplyDelete
  33. You have a good point here!I totally agree with what you have said!! Thanks for sharing your views. hope more people will read this article!!! tree removal services fort lauderdale

    ReplyDelete
  34. This comment has been removed by the author.

    ReplyDelete
  35. I think this is one of the most significant information for me. And i’m glad reading your article.
    tree trimming companies north lauderdale

    ReplyDelete
  36. This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post.tree trimming hallandale beach

    ReplyDelete
  37. Thanks for the wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading. tree trimming services oakland park fl

    ReplyDelete
  38. keep up the good work. this is an Assam post. this to helpful, i have reading here all post. i am impressed. thank you. this is our digital marketing training center. This is an online certificate course
    digital marketing training in bangalore | https://www.excelr.com/digital-marketing-training-in-bangalore

    ReplyDelete
  39. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously in their life, he/she can earn his living by doing blogging.thank you for thizs article. devops online training

    ReplyDelete
  40. אין ספק שזה אחד הנושאים המעניינים. תודה על השיתוף.
    טבעות אירוסין מעוצבות

    ReplyDelete
  41. פוסט נחמד. חייב לשתף עם העוקבים שלי
    מגשי אירוח מתוקים

    ReplyDelete
  42. A superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place. residential tree services margate

    ReplyDelete
  43. אהבתי מאוד את סגנון הכתיבה.
    פינת אוכל

    ReplyDelete
  44. סגנון כתיבה מרענן, תודה על השיתוף.
    פינות אוכל נפתחות

    ReplyDelete
  45. רציתי רק לשאול, אפשר לשתף את הפוסט בבלוג שלי?
    טיפולים קוסמטיים

    ReplyDelete
  46. This post is good enough to make somebody understand this amazing thing, and I’m sure everyone will appreciate this interesting things.tree services near me in weston fl

    ReplyDelete
  47. A superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place. tree care companies deerfield beach

    ReplyDelete
  48. תמשיכו בפרסום פוסטים מעניינים כמו זה. תודה.
    פלאנרז הפקות

    ReplyDelete
  49. Thanks for the wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading. emergency tree services sunrise fl

    ReplyDelete
  50. פוסט נחמד. חייב לשתף עם העוקבים שלי.
    שטיח לחדר ילדים

    ReplyDelete
  51. אין ספק שהפוסט הזה דורש שיתוף. תודה.
    נכסים מניבים

    ReplyDelete
  52. Great article and a nice way to promote online. I’m satisfied with the information that you provided palm tree services pompano beach

    ReplyDelete
  53. Excellent information Providing by your Article. Thanks
    cat hanging necklace

    ReplyDelete
  54. You have a good point here!I totally agree with what you have said!! Thanks for sharing your views. hope more people will read this article!!! commercial tree services parkland

    ReplyDelete
  55. This comment has been removed by the author.

    ReplyDelete
  56. I think this is one of the most significant information for me. And i’m glad reading your article. stump grinding lauderdale lakes

    ReplyDelete
  57. This is a great article thanks for sharing this informative information. I will visit your blog regularly for some latest post.stump removal hollywood fl

    ReplyDelete
  58. This comment has been removed by the author.

    ReplyDelete
  59. Amazing work. Extra-normal manner of taking pictures the details.
    Thanks for sharing. Waiting in your destiny updates.
    7starhd

    ReplyDelete
  60. Snapdeal Winner List 2020 here came up with an Offer where you can win Snapdeal prize list by just playing a game & win prizes.
    Snapdeal winner name also check the Snapdeal lucky draw

    ReplyDelete
  61. תמשיכו בפרסום פוסטים מעניינים כמו זה. תודה.
    מערכת מצלמות אבטחה

    ReplyDelete
  62. אין ספק שהפוסט הזה דורש שיתוף. תודה.
    ניהול מוניטין

    ReplyDelete
  63. אין ספק שזה אחד הנושאים המעניינים. תודה על השיתוף.
    הדפסת תמונה על בלוק עץ

    ReplyDelete
  64. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    data analytics courses in mumbai
    data science interview questions

    ReplyDelete
  65. I finally found great post here.I will get back here. I just added your blog to my bookmark sites. thanks.Quality posts is the crucial to invite the visitors to visit the web page, that's what this web page is providing.
    data science course Mumbai
    data science interview questions
    data analytics course in mumbai

    ReplyDelete
  66. snapdeal here thought of an Offer where you can win exceptional snapdeal prize by simply playing a game and win prizes Call @7596886123"
    Snapdeal winner 2020
    Snapdeal lucky draw tatasafari 2020
    Snapdeal lucky winner list 2020
    Snapdeal lucky draw 2020

    ReplyDelete
  67. We as a team of real-time industrial experience with a lot of knowledge in developing applications in python programming (7+ years) will ensure that we will deliver our best in python training in vijayawada. , and we believe that no one matches us in this context.

    ReplyDelete
  68. Snapdeal winner listhere an offer you can win the Snapdeal lucky draw.Here you can win the Snapdeal winner list 2020 and Snapdeal winner list. Snapdeal lucky drawwhere you can win the prize just playing the game.You can also check Snapdeal winner name and Snapdeal winner name 2020

    ReplyDelete
  69. Thanks for sharing such a wonderful blog on Amazon Web Services .
    This blog contains so much data about Amazon Web Services ,like if anyone who is searching for the Amazon Web Services data,They will easily grab the knowledge from this .Requested you to please keep sharing these type of useful content so that other can get benefit from your shared content.
    Thanks and Regards,
    Amazon Web Services training in Chennai
    Best Amazon Web Services training in chennai
    Top Amazon Web Services Training in chennai
    Amazon Web Services training fees in Velachery,Chennai

    ReplyDelete
  70. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck... Thank you!!! digital marketing courses in Bangalore

    ReplyDelete
  71. Shopclues lucky draw 2020| Shopclues winner 2020|Get to Know Shopclues Lucky Draw Winner 2020. Call Shopclues Helpline Phone Number+91-9835565523 for Shopclues Online Shopping Lucky Draw.
    Shopclues online lucky draw winner
    Shopclues winner 2020
    shopclues car lucky draw

    ReplyDelete
  72. Amazing blog, thanks for sharing this. OGEN Infosystem provides the best service for Website and SEO Services at a reasonable cost.
    SEO Agency in Noida

    ReplyDelete
  73. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sap bi training sharing this great content to my vision, keep sharing...

    ReplyDelete

  74. Pretty article! I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision. i also want to share some infor mation regarding sap pp online training and sap sd videos.keep sharing.

    ReplyDelete
  75. Snapdeal winner 2020 | Dear customer, you can complain here If you get to call and SMS regarding Snapdeal lucky draw, Mahindra xuv 500, lucky draw contest, contact us at to know the actual Snapdeal prize winners 2020.
    Snapdeal winner 2020
    Snapdeal lucky draw winner 2020
    Snapdeal lucky draw contest 2020
    snapdeal winner prizes 2020

    ReplyDelete

  76. This post is really nice and informative. The explanation given is really comprehensive and informative. I also want to say about the digital marketing course near me

    ReplyDelete
  77. Awesome article, it was exceptionally helpful! I simply began in this and I'm becoming more acquainted with it better. The post is written in very a good manner and it contains many useful information for me. Thank you very much and will look for more postings from you.


    digital marketing blog
    digital marketing bloggers
    digital marketing blogs
    digital marketing blogs in india
    digital marketing blog 2020
    digital marketing blog sites
    skartec's digital marketing blog
    skartec's blog
    digital marketing course
    digital marketing course in chennai
    digital marketing training
    skartec digital marketing academy

    ReplyDelete