Uploading a directory tree to AWS S3 in a few lines of Python
I occasionally need to upload a directory tree to an AWS S3 bucket, which can have many levels of subdirectories and files. This is doable with the Python boto library, but the last time I used it to do this task it was quite tedious.
import fs_s3fs from fs.copy import ( copy_dir as fs_copy_dir, ) from fs.walk import Walker as fs_walker fs = fs_s3fs.S3FS( "BUCKET", "ACCESS_KEY", "SECRET_KEY", "af-south-1", ) fs_copy_dir( "/", SRC_DIRECTORY, fs, TARGET_DIR, walker=fs_walker( exclude=["*.html"] ), )
The first argument to fs.copy_dir is the local filesystem, given as the root of a Linux or Docker filesystem in this case.