Wednesday, June 19, 2013

2 reasons why we select SimpleDB instead of DynamoDB

If you search on google with keywords "SimpleDB vs DynamoDB", there will be a lot of helpful posts. Most of them give you 3 to 7 reasons to select DynamoDB. However, today I'll share some experience of using SimpleDB instead of DynamoDB.

I got some issues when use DynamoDB in my production, and finally found that SimpleDB is fit in my case perfectly. I think the choice of SimpleDB and DynamoDB should NOT rely on the performance or the benefits of the DynamoDB/SimpleDB, instead, based on the limitation and real requirement in my product.

Some background: I have some data previously saved in MongoDB, the amount of data will mostly not exceed 2G bytes in SimpleDB. Now we decided not to maintain our MongoDB database servers, but leverage AWS SimpleDB or DynamoDB to reduce the cost on ops.

Both SimpleDB/DynamoDB is key/value pair database. There are some workaround to store a JSON document, but will introduce additional cost. The data structure in my MongoDB is not too complicated and can be convert to key-value pair. So, before you choose SimpleDB or DynamoDB as your database backend, you must understand this fundamental.

Reason 1: Not flexible on indexing. With DynamoDB you have to set indexing fields before creating the database, and cannot be modified. This is really limited the future change. DynamoDB supports 2 mode of data lookup, "Query" and "Scan". "Query": based on hash key and secondary keys, high performance. However, when you query data, “hash” key must be set. For example, suppose we have “id” key as hash key. When query by “id”, it’s good, we can get best performance. But when we query only by a field "name", we have to shift to “Scan” because hash key is not used. The performance of "Scan" is totally not acceptable because AWS will scan every record. I created a sample DynamoDb with 100,000 records, and each record has 6 fields. With "Scan", it costs 2 ~ 6 minutes to selecting ONE record by adding condition on one field. Here is the testing code in Java:

DynamoDBScanExpression scan = new DynamoDBScanExpression();

scan.addFilterCondition("count", new Condition().withAttributeValueList(new AttributeValue().withN("70569")).withComparisonOperator(ComparisonOperator.EQ));

System.out.println("1=> " + new Date());

PaginatedScanList<Book> list = mapper.scan(Book.class, scan);

System.out.println("2=> " + new Date());

Object[] all = list.toArray();

System.out.println(all.length); // should be 1

System.out.println("3=> " + new Date()); // 2 ~ 6 minutes comparing to date after “2=>”, in most cases around 2 minutes

SimpleDB does not have this limitations. SimpleDB create index for "EVERY" field in a table(actually AWS use the term "domain", and MongoDB use "collection"). I modified a little bit the code and test on SimpleDB, here are the results:

  • Query 500 (use "limits" to get the first 500 items in a “select” call) items with no condition: about 400 ms to complete. The sample application running on my local machine. If it is running on EC2, it should be within 100 ms. 
  • Query 500 items with 1 condition, also about 400 ms to complete.
Reason 2: Not cost effective for our case. The DynamoDB charge money by capacity of Read/Writes per seconds. Please note that the capacity is based on read/write your records instead of the read/write API call, and no matter you use batch or not. Here are more details in my test.  I used batch API to send 1000 records with more than 1000 bytes for each record. There will cost 50 seconds to finish the batch when the write capacity was set to 20/seconds. While I keep the my application running, and change the capacity on AWS console to 80/seconds, there will take 12 to 25 seconds to complete one batch(ideally it should be 1000/80 = 12.5 seconds, the extra time comes from network latency because I’m sending more than 1 megabytes data per API call). 

In our case, we may read the 500 records in SimpleDB into memory, but read nothing in next 10 minutes. With SimpleDB we can complete it in 500 milliseconds. With DynamoDB we have to set read capacity to 1000 reads/seconds, and it will cost $94.46 per month(via AWS Simple Monthly Calculator). With SimpleDB, it may cost less than 1 dollar.

Conclusion: DynamoDB is really designed for high performance database. SimpleDB has more flexibility. Here what I mean "really designed for high performance" to DynamoDB is, if you choose DynamoDB, you must make sure you have well designed your architecture for high traffic dynamic content. If you have design your architecture targeting high traffic dynamic content and high performance, DynamoDB may perfectly match your request. In our case, SimpleDB is enough, excellent flexibility, and cost effective. Before looking for the comparison of SimpleDB and DynamoDB, design your architecture first. DynamoDB is good, but not fit for everyone.

Here are some useful links:


  1. Nice to see a post like this.

    Reason 1: Not flexible on indexing.

    I totally agree that DynamoDB is not flexible here. You can not add on or delete indexes. For your use case, seems like making "id" the hashkey and "name" as the range key, maybe adding some other columns as local secondary index as well. But anyway, if your usage changes again you would need to nuke your table and re-create table and schema - yuck!

    Reason 2: Not cost effective for our case.
    Hmm....this sucks too. One possible thing is, have you thought about dialing up read capacity in the morning and dialing down it every night?

    1. Great Article Cloud Computing Projects

      Networking Projects

      Final Year Projects for CSE

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

  2. I am also agree with these two points. I am adding three more points in the comparison -

    1. Amazon SimpleDB offers simplicity and flexibility whereas Amazon DynamoDB offers good performance and incremental scalability.
    2. Amazon SimpleDB pricing is based on your actual box usage whereas DynamoDB is priced according to how much request capacity you have requested.
    3. Amazon SimpleDB can be useful for those who need a non-relational database for storage of smaller, non-structural data whereas Amazon DynamoDB can be useful for those who need a fast, highly scalable non-relational database.

  3. Interesting post, thanks for sharing your experience

  4. Very nice post, thanks!! I'm using simpledb for months now with single domain and I don't have any problem however, for us to scale we might need to use more than 1 domain. Example, we'll be using 2 domains to store personal data of users. data will be randomly saved to both domain, my problem for now is with sorting of data with 2 or more domains. pls help if you've encounter the same scenario. Thank you.

    1. This comment has been removed by the author.

  5. Nice post. Thanks for sharing.

  6. Great post. Thanks for sharing. I was just thinking about SimpleDB and this gives us some good points to go for it.

  7. Very good information. we need learn with real time examples and for this we choose good training institute, who were interested to know about AWS which is quite interesting. We need a good training institute for my learning .. so people making use of the free demo classes.
    Many training institute provides free demo classes. One of the best training institute in Bangalore is Apponix Technologies.


  8. الرائد افضل شركات تنظيف خزانات المياه يسعدنا ان نقدم لكم افضل خدمات
    شركة غسيل خزانات بالمدينة المنورة تنظيف خزانات بالمدينة المنورة
    افضل شركة تنظيف منازل بالمدينة المنورة شركة تنظيف بيوت بالمدينة المنورة

  9. It’s hard to come by experienced people about this subject, but you seem like you know what you’re talking about! Thanks.
    Java Training in Bangalore
    Python Training In Bangalore

  10. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.

  11. The information that you have shared is really useful for everyone. wonderful article. Nicely written and great info.Thanks to share the more information's.PHP Training in Chennai

    PHP Online Training in Chennai

    Machine Learning Training in Chennai

    iOT Training in Chennai

    Blockchain Training in Chennai

    Open Stack Training in Chennai

  12. The great thing about this post is quality information. I always like to read amazingly useful and quality content. Your article is amazing, thank you for sharing this article.
    tata cliq offers
    mi offers
    swiggy offers
    zomato coupons

  13. I unquestionably appreciate each and every piece of it. It is an extraordinary site and has a pleasant offer. I need to bless your heart. Great job! You all do an incredible blog, and have some extraordinary substance. Keep doing awesome. Oracle Fusion Cloud Manufacturing training Course

  14. Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good work.
    data science using python and r programming Hyderabad

  15. I just loved your article on the beginners guide to starting a blog.If somebody take this blog article seriously
    in their life, he/she can earn his living by doing blogging.Thank you for this article.
    top tibco sportfire online training

  16. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environment react course in vijayawada. ,

  17. Useful info. Fortunate me I found your website accidentally, and I’m stunned why this twist of fate did not happened earlier! I bookmarked it.
    BY COGNEX IS THE AWS Training in Chennai

  18. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one
    Best Data Science Courses in Hyderabad

  19. Nice & Informative Blog !
    QuickBooks is an accounting software that has rapidly captured the global market,QuickBooks Error 8007 while working on this software. To get rid of such problems, call us at 1-855-977-7463 and get immediate support for QuickBooks issues.

  20. You have done a great job. I will definitely dig it and personally recommend to my friends. I am confident they will be benefited from this site. rizbuz

  21. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    data science in bangalore

  22. Thanks for spending all your pleasant time to make such a Creative content for us. AWS course in Chennai

  23. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Course in Bangalore

  24. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    best data science courses in bangalore