• duuyidong@gmail.com

Speed Up Your AWS S3 Client

Our team recently had a performance issue with some data processing. Every day we have 24 files of 30GB generated in S3, and we’re having a Fargate cluster to download and process those data, it takes 12 hours to processing all 600+GB files, which is too slow as we want to increase the size of file for processing. After a serial of improvement, we successfully reduce the processing to 1.5 hours.

This is a sample project to explain what improvement we’ve done: https://github.com/ADU-21/s3-parallel-download



Read more

AWS DynamoDB Study Note

Relational Databases is friendly to DDD(Domain Driven Development), provide point in time of the transactions to ensure consistency, however, it can be costly to do horizontal scale(read/write replications) and vertical scale(schema changes). That’s why we some time need NoSQL to save some money in business scenarios that do not require transactions and are not sensitive to consistency.

DynamoDB was announced by Amazon CTO Werner Vogels on in 2012, 14 years after NoSQL was proposed in 1998. It supports key-value and document-oriented structure storage.



Read more

AWS Step Function

Our client recently has a deployment system that has been in use for more than 10 years and wants to migrate to the cloud, this blog shows how we migrate it step by step from a huge single application to serverless by AWS Step funtion.



Read more

AWS Data Analytics

Big things, fast, minimal set up, maximum security, low cost.



Read more

How to Code Python

Python is my favorite language because it’s simple and fast. It has been 3 years since first time I learn python, I’ve wrote a lot of article about python inital in my first year coiding python, feel free to read it if you understand chinese well.

this article will focus on engineering practice of python and include an example about how to build a tool with python.



Read more

My Name is Luke, come from a story

It has been long time, haven’t think about Jackie Chan and the movice I have been made with him. the Skiptrace, Can’t see it’s a very successful movie, but it’s do help me growth every much.



Read more

Welcome To Hexo

Welcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on github.


Read more