Finally figured out why some of my expired items fail to delete when expected. (bold mine)
DynamoDB typically deletes expired items within 48 hours of expiration. The exact duration within which an item truly gets deleted after expiration is specific to the nature of the workload and the size of the table. Items that have expired and not been deleted will still show up in reads, queries, and scans. These items can still be updated and successful updates to change or remove the expiration attribute will be honored.
Bottom line: you can't rely on DynamoDB's
ttl column based expiration to do minute-precise expiration of things in your serverless app. Luckily for my purposes, I don't need minute precision. I do however need hour precision, so I'm planning to write a little scheduled lambda that kicks off every sixty minutes and reaps whatever records have expired but not deleted yet, and delete them manually.
Purchase my Serverless book at https://leanpub.com/serverless