Wednesday, June 19, 2013

2 reasons why we select SimpleDB instead of DynamoDB

If you search on google with keywords "SimpleDB vs DynamoDB", there will be a lot of helpful posts. Most of them give you 3 to 7 reasons to select DynamoDB. However, today I'll share some experience of using SimpleDB instead of DynamoDB.

I got some issues when use DynamoDB in my production, and finally found that SimpleDB is fit in my case perfectly. I think the choice of SimpleDB and DynamoDB should NOT rely on the performance or the benefits of the DynamoDB/SimpleDB, instead, based on the limitation and real requirement in my product.

Some background: I have some data previously saved in MongoDB, the amount of data will mostly not exceed 2G bytes in SimpleDB. Now we decided not to maintain our MongoDB database servers, but leverage AWS SimpleDB or DynamoDB to reduce the cost on ops.

Both SimpleDB/DynamoDB is key/value pair database. There are some workaround to store a JSON document, but will introduce additional cost. The data structure in my MongoDB is not too complicated and can be convert to key-value pair. So, before you choose SimpleDB or DynamoDB as your database backend, you must understand this fundamental.

Reason 1: Not flexible on indexing. With DynamoDB you have to set indexing fields before creating the database, and cannot be modified. This is really limited the future change. DynamoDB supports 2 mode of data lookup, "Query" and "Scan". "Query": based on hash key and secondary keys, high performance. However, when you query data, “hash” key must be set. For example, suppose we have “id” key as hash key. When query by “id”, it’s good, we can get best performance. But when we query only by a field "name", we have to shift to “Scan” because hash key is not used. The performance of "Scan" is totally not acceptable because AWS will scan every record. I created a sample DynamoDb with 100,000 records, and each record has 6 fields. With "Scan", it costs 2 ~ 6 minutes to selecting ONE record by adding condition on one field. Here is the testing code in Java:

DynamoDBScanExpression scan = new DynamoDBScanExpression();

scan.addFilterCondition("count", new Condition().withAttributeValueList(new AttributeValue().withN("70569")).withComparisonOperator(ComparisonOperator.EQ));

System.out.println("1=> " + new Date());

PaginatedScanList<Book> list = mapper.scan(Book.class, scan);

System.out.println("2=> " + new Date());

Object[] all = list.toArray();

System.out.println(all.length); // should be 1

System.out.println("3=> " + new Date()); // 2 ~ 6 minutes comparing to date after “2=>”, in most cases around 2 minutes

SimpleDB does not have this limitations. SimpleDB create index for "EVERY" field in a table(actually AWS use the term "domain", and MongoDB use "collection"). I modified a little bit the code and test on SimpleDB, here are the results:

  • Query 500 (use "limits" to get the first 500 items in a “select” call) items with no condition: about 400 ms to complete. The sample application running on my local machine. If it is running on EC2, it should be within 100 ms. 
  • Query 500 items with 1 condition, also about 400 ms to complete.
Reason 2: Not cost effective for our case. The DynamoDB charge money by capacity of Read/Writes per seconds. Please note that the capacity is based on read/write your records instead of the read/write API call, and no matter you use batch or not. Here are more details in my test.  I used batch API to send 1000 records with more than 1000 bytes for each record. There will cost 50 seconds to finish the batch when the write capacity was set to 20/seconds. While I keep the my application running, and change the capacity on AWS console to 80/seconds, there will take 12 to 25 seconds to complete one batch(ideally it should be 1000/80 = 12.5 seconds, the extra time comes from network latency because I’m sending more than 1 megabytes data per API call). 

In our case, we may read the 500 records in SimpleDB into memory, but read nothing in next 10 minutes. With SimpleDB we can complete it in 500 milliseconds. With DynamoDB we have to set read capacity to 1000 reads/seconds, and it will cost $94.46 per month(via AWS Simple Monthly Calculator). With SimpleDB, it may cost less than 1 dollar.

Conclusion: DynamoDB is really designed for high performance database. SimpleDB has more flexibility. Here what I mean "really designed for high performance" to DynamoDB is, if you choose DynamoDB, you must make sure you have well designed your architecture for high traffic dynamic content. If you have design your architecture targeting high traffic dynamic content and high performance, DynamoDB may perfectly match your request. In our case, SimpleDB is enough, excellent flexibility, and cost effective. Before looking for the comparison of SimpleDB and DynamoDB, design your architecture first. DynamoDB is good, but not fit for everyone.

Here are some useful links:

57 comments:

  1. Nice to see a post like this.

    Reason 1: Not flexible on indexing.

    I totally agree that DynamoDB is not flexible here. You can not add on or delete indexes. For your use case, seems like making "id" the hashkey and "name" as the range key, maybe adding some other columns as local secondary index as well. But anyway, if your usage changes again you would need to nuke your table and re-create table and schema - yuck!

    Reason 2: Not cost effective for our case.
    Hmm....this sucks too. One possible thing is, have you thought about dialing up read capacity in the morning and dialing down it every night?



    ReplyDelete
    Replies
    1. Great Article Cloud Computing Projects

      Networking Projects

      Final Year Projects for CSE

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. I am also agree with these two points. I am adding three more points in the comparison -

    1. Amazon SimpleDB offers simplicity and flexibility whereas Amazon DynamoDB offers good performance and incremental scalability.
    2. Amazon SimpleDB pricing is based on your actual box usage whereas DynamoDB is priced according to how much request capacity you have requested.
    3. Amazon SimpleDB can be useful for those who need a non-relational database for storage of smaller, non-structural data whereas Amazon DynamoDB can be useful for those who need a fast, highly scalable non-relational database.

    ReplyDelete
  3. Interesting post, thanks for sharing your experience

    ReplyDelete
  4. Very nice post, thanks!! I'm using simpledb for months now with single domain and I don't have any problem however, for us to scale we might need to use more than 1 domain. Example, we'll be using 2 domains to store personal data of users. data will be randomly saved to both domain, my problem for now is with sorting of data with 2 or more domains. pls help if you've encounter the same scenario. Thank you.

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  5. Nice post. Thanks for sharing.

    ReplyDelete
  6. Great post. Thanks for sharing. I was just thinking about SimpleDB and this gives us some good points to go for it.

    ReplyDelete
  7. Very good information. we need learn with real time examples and for this we choose good training institute, who were interested to know about AWS which is quite interesting. We need a good training institute for my learning .. so people making use of the free demo classes.
    Many training institute provides free demo classes. One of the best training institute in Bangalore is Apponix Technologies.
    https://www.apponix.com/AWS-Essentials-Solutions-Architect/AWS-solution-arcchitect-training-in-bangalore.html

    ReplyDelete

  8. الرائد افضل شركات تنظيف خزانات المياه يسعدنا ان نقدم لكم افضل خدمات
    شركة غسيل خزانات بالمدينة المنورة تنظيف خزانات بالمدينة المنورة
    افضل شركة تنظيف منازل بالمدينة المنورة شركة تنظيف بيوت بالمدينة المنورة

    ReplyDelete
  9. It’s hard to come by experienced people about this subject, but you seem like you know what you’re talking about! Thanks.
    Java Training in Bangalore
    Python Training In Bangalore

    ReplyDelete
  10. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.
    360DigiTMG

    ReplyDelete
  11. The information that you have shared is really useful for everyone. wonderful article. Nicely written and great info.Thanks to share the more information's.PHP Training in Chennai

    PHP Online Training in Chennai

    Machine Learning Training in Chennai

    iOT Training in Chennai

    Blockchain Training in Chennai

    Open Stack Training in Chennai

    ReplyDelete
  12. The great thing about this post is quality information. I always like to read amazingly useful and quality content. Your article is amazing, thank you for sharing this article.
    tata cliq offers
    mi offers
    swiggy offers
    zomato coupons

    ReplyDelete
  13. I unquestionably appreciate each and every piece of it. It is an extraordinary site and has a pleasant offer. I need to bless your heart. Great job! You all do an incredible blog, and have some extraordinary substance. Keep doing awesome. Oracle Fusion Cloud Manufacturing training Course

    ReplyDelete
  14. Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good work.
    data science using python and r programming Hyderabad

    ReplyDelete
  15. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
    in their life, he/she can earn his living by doing blogging.Thank you for this article.
    top tibco sportfire online training

    ReplyDelete
  16. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environment react course in vijayawada. ,

    ReplyDelete
  17. Useful info. Fortunate me I found your website accidentally, and I’m stunned why this twist of fate did not happened earlier! I bookmarked it.
    BY COGNEX IS THE AWS Training in Chennai

    ReplyDelete
  18. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one
    Best Data Science Courses in Hyderabad

    ReplyDelete
  19. Nice & Informative Blog !
    QuickBooks is an accounting software that has rapidly captured the global market,QuickBooks Error 8007 while working on this software. To get rid of such problems, call us at 1-855-977-7463 and get immediate support for QuickBooks issues.

    ReplyDelete
  20. You have done a great job. I will definitely dig it and personally recommend to my friends. I am confident they will be benefited from this site. rizbuz

    ReplyDelete
  21. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data science in bangalore

    ReplyDelete
  22. Thanks for spending all your pleasant time to make such a Creative content for us. AWS course in Chennai

    ReplyDelete
  23. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Course in Bangalore

    ReplyDelete
  24. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    best data science courses in bangalore

    ReplyDelete
  25. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    data scientist course in bangalore

    ReplyDelete
  26. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    data analytics course in bangalore

    ReplyDelete
  27. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data analytics courses in bangalore

    ReplyDelete
  28. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    data scientist course in bangalore

    ReplyDelete
  29. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data science course in chennai

    ReplyDelete
  30. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
    Data Science Course Syllabus

    ReplyDelete
  31. I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
    data analytics training in bangalore

    ReplyDelete
  32. This post is very simple to read and appreciate without leaving any details out. Great work!
    business analytics course

    ReplyDelete
  33. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    artificial intelligence training in chennai

    ReplyDelete
  34. Thanks for posting the best information and the blog is very important.data science institutes in hyderabad

    ReplyDelete
  35. Impressive blog to be honest definitely this post will inspire many more upcoming aspirants. Eventually, this makes the participants to experience and innovate themselves through knowledge wise by visiting this kind of a blog. Once again excellent job keep inspiring with your cool stuff.

    data science certification in bangalore

    ReplyDelete
  36. Fantastic article I ought to say and thanks to the info. Instruction is absolutely a sticky topic. But remains one of the top issues of the time. I love your article and look forward to more.
    Data Science Course in Bangalore

    ReplyDelete
  37. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
    Data Science Training in Chennai

    ReplyDelete
  38. Impressive. Your story always brings hope and new energy. Keep up the good work.
    business analytics course

    ReplyDelete
  39. Thank you quite much for discussing this type of helpful informative article. Will certainly stored and reevaluate your Website.
    Data Analytics Course in Bangalore

    ReplyDelete
  40. Very wonderful informative article. I appreciated looking at your article. Very wonderful reveal. I would like to twit this on my followers. Many thanks! .
    Data Analytics training in Bangalore

    ReplyDelete
  41. Nice to be seeing your site once again, it's been weeks for me. This article which ive been waited for so long. I need this guide to complete my mission inside the school, and it's same issue together along with your essay. Thanks, pleasant share.
    Data Science training in Bangalore

    ReplyDelete
  42. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    artificial intelligence training in chennai

    ReplyDelete
  43. Awesome article. I enjoyed reading your articles. this can be really a good scan for me. wanting forward to reading new articles. maintain the nice work!

    DevOps Training in Hyderabad

    ReplyDelete
  44. Wonderful article, thanks for putting this together! This is obviously one great post. Thanks for the valuable information and insights you have so provided here...

    DevOps Training in Hyderabad

    ReplyDelete
  45. Terrific post thoroughly enjoyed reading the blog and more over found to be the tremendous one. In fact, educating the participants with it's amazing content. Hope you share the similar content consecutively.

    data science course in varanasi

    ReplyDelete
  46. What a really awesome post this is. Truly, one of the best posts I've ever witnessed to see in my whole life. Wow, just keep it up.
    digital marketing courses in hyderabad with placement

    ReplyDelete
  47. Very wonderful informative article. I appreciated looking at your article. Very wonderful reveal. I would like to twit this on my followers. Many thanks! .
    Data Analytics training in Bangalore

    ReplyDelete