You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
**Sanhe Hu** is a very active **Python Developer** Since 2010. Research area includes **Machine Learning, Big Data Infrastructure, Block Chain, Business Intelligent, AWS, Distributive System**. Love photography, outdoor, arts, game, and also the best `Python <https://www.python.org/>`_.
5
+
**Sanhe Hu** is a very active **Python Developer and AWS veteran** Since 2010. Research area includes **Machine Learning, Big Data Infrastructure, Block Chain, Business Intelligent, AWS, Distributive System**. Love photography, outdoor, arts, game, and also the best `Python <https://www.python.org/>`_.
``pynamodb_mate`` provides advanced best practice using DynamoDB in python. Built on top of `pynamodb <https://pynamodb.readthedocs.io/en/latest/>`_ python library. It maintain the compatibility to major version of ``pynamodb`` library. For example ``pynamodb_mate>=5.0.0,<6.0.0`` is compatible to ``pynamodb>=5.0.0,<6.0.0``.
66
+
``pynamodb_mate`` provides advanced best practice using DynamoDB in python. Built on top of `pynamodb <https://pynamodb.readthedocs.io/en/latest/>`_ python library. It maintain the compatibility to major version of ``pynamodb`` library. For example ``pynamodb_mate>=5.0.0,<6.0.0`` is compatible to ``pynamodb>=5.0.0,<6.0.0``, ``pynamodb_mate==5.5.1.X`` is compatible to ``pynamodb>=5.5.1,<6.0.0``, ``pynamodb_mate==6.0.0.X`` is compatible to ``pynamodb>=6.0.0,<7.0.0``.
65
67
66
68
67
69
Disclaimer
@@ -71,21 +73,15 @@ Even though the author is a Dynamodb Subject Matter Expert from AWS, but this pr
Click on the link below to see detailed tutorial and examples.
75
-
76
-
- ⭐ `Store Large Object in DynamoDB <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/Store-Large-Object-in-DynamoDB.ipynb>`_: This feature allows you to store any Python object and arbitrary big data in DynamoDB that can exceed the 400KB limit.
77
-
- ⭐ `Client Side Encryption <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/Client-Side-Encryption.ipynb>`_: This feature allows you to use your own encryption key to encrypt your data before it is sent to the DynamoDB.
78
-
- ⭐ `Compressed Attribute <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/Compressed-Attribute.ipynb>`_: This feature can automatically compress the data before it is sent to the DynamoDB.
79
-
- ⭐ `AWS DynamoDB Console Url <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/AWS-DynamoDB-Console-Url.ipynb>`_: This feature can print the AWS DynamoDB console url for the table, items. You can use this in you log to quickly jump to the console to debug.
``pynamodb_mate`` also provides some commonly used patterns. It is based on the author's experience providing solutions to many customers from different industries. Click on the link below to see detailed tutorial and examples.
85
-
86
-
- 🎉 `Cache <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/patterns/cache.ipynb>`_: This feature gives you a "Redis" liked, serverless, Zero-ops, auto-scaling, high performance, pay-as-you-go cache layer based on DynamoDB.
87
-
- 🎉 `Status Tracker <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/patterns/status-tracker.ipynb>`_: This feature gives you the ability to track status of each task, and error-handling, retry, concurrency control out-of-the-box.
88
-
- 🎉 `Relationship <https://github.com/MacHu-GWU/pynamodb_mate-project/blob/master/examples/patterns/relationship.ipynb>`_: This feature allow you to manage mass amount entity and one-to-many, many-to-many relationship in DynamoDB using the ultimate data modeling strategy.
76
+
``pynamodb_mate`` provides some commonly used patterns. It is based on the author's experience providing solutions to many customers from different industries. Click on the link below to see detailed tutorial and examples.
77
+
78
+
- ⭐ `Store Large Object in DynamoDB <https://pynamodb-mate.readthedocs.io/en/latest/01-Store-Large-Object-in-DynamoDB/index.html>`_: This feature allows you to store any Python object and arbitrary big data in DynamoDB that can exceed the 400KB limit. It also handles data consistency and data integrity between DynamoDB and S3.
79
+
- ⭐ `Client Side Encryption <https://pynamodb-mate.readthedocs.io/en/latest/02-Client-Side-Encryption/index.html>`_: This feature allows you to use your own encryption key to encrypt your data before it is sent to the DynamoDB.
80
+
- ⭐ `Compressed Attribute <https://pynamodb-mate.readthedocs.io/en/latest/03-Compressed-Attribute/index.html>`_: This feature can automatically compress the data before it is sent to the DynamoDB, it would save you a lot of money if you use JSON attribute in DynamoDB.
81
+
- ⭐ `AWS DynamoDB Console Url <https://pynamodb-mate.readthedocs.io/en/latest/04-DynamoDB-Consule-Url/index.html>`_: This feature can print the AWS DynamoDB console url for the table, items. You can use this in you log to quickly jump to the console to debug.
82
+
- ⭐ `Use DynamoDB as a Cache Backend <https://pynamodb-mate.readthedocs.io/en/latest/05-Use-DynamoDB-as-Cache-Backend/index.html>`_: This feature gives you a "Redis" liked, serverless, Zero-ops, auto-scaling, high performance, pay-as-you-go cache layer based on DynamoDB.
83
+
- ⭐ `Enable status tracking for business critical application using Amazon DynamoDB <https://pynamodb-mate.readthedocs.io/en/latest/06-Status-Tracker/index.html>`_: This feature gives you the ability to track status of each task, and error-handling, retry, concurrency control out-of-the-box.
84
+
- ⭐ `Modeling Relational Data in DynamoDB <https://pynamodb-mate.readthedocs.io/en/latest/07-Modeling-Relational-Data-in-DynamoDB/index.html>`_: This feature allow you to manage mass amount entity and one-to-many, many-to-many relationship in DynamoDB using the ultimate data modeling strategy that has high performance and scales well. Made data modeling in DynamoDB deadly simple.
Copy file name to clipboardexpand all lines: docs/source/03-Compressed-Attribute/index.ipynb
+1-1
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@
14
14
"source": [
15
15
"# Compressed Attribute\n",
16
16
"\n",
17
-
"## Summary\n",
17
+
"## Overview\n",
18
18
"\n",
19
19
"Sometimes you want to compress the data before store to save DB space. For example, in an E-commerce data model, an order has many items like this: ``[{\"item_name\": \"apple\", \"item_count\": 12}, {\"item_name\": \"banana\", \"item_count\": 5}]``. There are lots of repeated information such as the keys ``\"item_name\"`` and ``\"item_count\"``.\n",
Copy file name to clipboardexpand all lines: docs/source/06-Status-Tracker/index.ipynb
+7-7
Original file line number
Diff line number
Diff line change
@@ -43,7 +43,7 @@
43
43
}
44
44
},
45
45
"source": [
46
-
"# Declare Your DynamoDB Status Tracking Table\n",
46
+
"## Declare Your DynamoDB Status Tracking Table\n",
47
47
"\n",
48
48
"1. First, We define some status code using the enum Python standard library. It improves the code readability and avoids hard coding meaningless integers everywhere in the code base.\n",
49
49
"2. We declare a DynamoDB table data model and the status GSI index schema.\n",
@@ -193,7 +193,7 @@
193
193
}
194
194
},
195
195
"source": [
196
-
"# Initialize a Task\n",
196
+
"## Initialize a Task\n",
197
197
"\n",
198
198
"The ``Tracker.new(task_id, data)`` method can be used to initialize a task and save to DynamoDB using the ``DEFAULT_STATUS``."
199
199
]
@@ -402,7 +402,7 @@
402
402
}
403
403
},
404
404
"source": [
405
-
"# Error Handling\n",
405
+
"## Error Handling\n",
406
406
"\n",
407
407
"Let's reset the task and do it one more time, this time the job logic will fail.\n",
408
408
"\n",
@@ -592,7 +592,7 @@
592
592
}
593
593
},
594
594
"source": [
595
-
"# Ignore Task if Failed Too Many Times\n",
595
+
"## Ignore Task if Failed Too Many Times\n",
596
596
"\n",
597
597
"You don't want a task that logically can never succeed to fail into a endless loop. In this example, we defined the max retry times is 3. If it failed 3 times in a row, it will be ignored. And if you want to start a task that is ignored, you will see an ``TaskIgnoredError``"
598
598
]
@@ -1079,7 +1079,7 @@
1079
1079
}
1080
1080
},
1081
1081
"source": [
1082
-
"# Save Custom Metadata to the Tracker\n",
1082
+
"## Save Custom Metadata to the Tracker\n",
1083
1083
"\n",
1084
1084
"You may want to store custom metadata to DynamoDB. Since DynamoDB is schemaless, you can store arbitrary data in DynamoDB."
1085
1085
]
@@ -1173,7 +1173,7 @@
1173
1173
}
1174
1174
},
1175
1175
"source": [
1176
-
"# Query Tasks by Status\n",
1176
+
"## Query Tasks by Status\n",
1177
1177
"\n",
1178
1178
"To restart some tasks from the last failed, you need to be able to query the tasks by status. The ``Tracker.query_by_status()`` method allow you to get tasks by one or many status codes. By default, it returns tasks ordered by ``update_time``."
1179
1179
]
@@ -1480,7 +1480,7 @@
1480
1480
}
1481
1481
},
1482
1482
"source": [
1483
-
"# Conclusion\n",
1483
+
"## Conclusion\n",
1484
1484
"\n",
1485
1485
"In general, this solution can improve the visibility, resilience and reliability of a business critical application. There’s no upfront effort to use this solution, because DynamoDB is a fully managed service. Naturally, it is scalable to adapt to very high workload or unpredictable workload.\n",
Copy file name to clipboardexpand all lines: docs/source/07-Modeling-Relational-Data-in-DynamoDB/index.ipynb
+14-14
Original file line number
Diff line number
Diff line change
@@ -61,7 +61,7 @@
61
61
}
62
62
},
63
63
"source": [
64
-
"# Declare Entity Table and Lookup Index"
64
+
"## Declare Entity Table and Lookup Index"
65
65
]
66
66
},
67
67
{
@@ -113,7 +113,7 @@
113
113
}
114
114
},
115
115
"source": [
116
-
"# Declare Data Model for Each Entity\n",
116
+
"## Declare Data Model for Each Entity\n",
117
117
"\n",
118
118
"In this ultimate data modeling strategy, all entities are stored within a single table. To enhance code organization, readability, and maintainability, it is highly recommended to define subclasses for each entity type. By creating separate subclasses, you can encapsulate entity-specific logic, attributes, and behavior within their respective classes.\n",
119
119
"\n",
@@ -186,7 +186,7 @@
186
186
}
187
187
},
188
188
"source": [
189
-
"# Declare Data Model for Each Relationship\n",
189
+
"## Declare Data Model for Each Relationship\n",
190
190
"\n",
191
191
"Similarly, a relationship is just a DynamoDB item. You also need a class for each relationship."
192
192
]
@@ -299,7 +299,7 @@
299
299
}
300
300
},
301
301
"source": [
302
-
"# Declare Type Hint and Iterator Proxy Object\n",
302
+
"## Declare Type Hint and Iterator Proxy Object\n",
303
303
"\n",
304
304
"This is recommended but optional."
305
305
]
@@ -403,7 +403,7 @@
403
403
}
404
404
},
405
405
"source": [
406
-
"# Declare Relationship Settings\n",
406
+
"## Declare Relationship Settings\n",
407
407
"\n",
408
408
"The ``RelationshipSetting`` object is a central component of the library that stores all the metadata related to entity relationships. It serves as a configuration object that defines the characteristics and behavior of entities and their relationships.\n",
409
409
"\n",
@@ -837,7 +837,7 @@
837
837
}
838
838
},
839
839
"source": [
840
-
"# Setup Local DynamoDB Mock "
840
+
"## Setup Local DynamoDB Mock "
841
841
]
842
842
},
843
843
{
@@ -874,7 +874,7 @@
874
874
}
875
875
},
876
876
"source": [
877
-
"# Create Some Test Data"
877
+
"## Create Some Test Data"
878
878
]
879
879
},
880
880
{
@@ -982,7 +982,7 @@
982
982
}
983
983
},
984
984
"source": [
985
-
"# Declare Some Helper Functions For Testing"
985
+
"## Declare Some Helper Functions For Testing"
986
986
]
987
987
},
988
988
{
@@ -1108,7 +1108,7 @@
1108
1108
}
1109
1109
},
1110
1110
"source": [
1111
-
"# Test Video Ownership Relationship (One to Many)"
1111
+
"## Test Video Ownership Relationship (One to Many)"
1112
1112
]
1113
1113
},
1114
1114
{
@@ -1181,7 +1181,7 @@
1181
1181
}
1182
1182
},
1183
1183
"source": [
1184
-
"# Test Channel Ownership Relationship (One to Many)"
1184
+
"## Test Channel Ownership Relationship (One to Many)"
1185
1185
]
1186
1186
},
1187
1187
{
@@ -1285,7 +1285,7 @@
1285
1285
}
1286
1286
},
1287
1287
"source": [
1288
-
"# Test Channel and Video Relationship (Many to Many)"
1288
+
"## Test Channel and Video Relationship (Many to Many)"
1289
1289
]
1290
1290
},
1291
1291
{
@@ -1483,7 +1483,7 @@
1483
1483
}
1484
1484
},
1485
1485
"source": [
1486
-
"# Test Playlist and Video Relationship (Many to Many)"
1486
+
"## Test Playlist and Video Relationship (Many to Many)"
1487
1487
]
1488
1488
},
1489
1489
{
@@ -1681,7 +1681,7 @@
1681
1681
}
1682
1682
},
1683
1683
"source": [
1684
-
"# Test Youtuber Subscription Relationship (Many to Many)"
1684
+
"## Test Youtuber Subscription Relationship (Many to Many)"
1685
1685
]
1686
1686
},
1687
1687
{
@@ -1937,7 +1937,7 @@
1937
1937
}
1938
1938
},
1939
1939
"source": [
1940
-
"# Test Channel Subscription Relationship (Many to Many)"
1940
+
"## Test Channel Subscription Relationship (Many to Many)"
0 commit comments