Uploading a directory tree to AWS S3 in a few lines of Python

I occasionally need to upload a directory tree to an AWS S3 bucket, which can have many levels of subdirectories and files. This is doable with the Python boto library, but the last time I used it to do this task it was quite tedious.

Will McGugan’s excellent PyFilesystem and its S3FS interface reduces this task to a few lines. Here’s a minimalish but functional example (with bonus .html file exclusion):

import fs_s3fs
from fs.copy import (
    copy_dir as fs_copy_dir,
)
from fs.walk import Walker as fs_walker

fs = fs_s3fs.S3FS(
    "BUCKET",
    "ACCESS_KEY",
    "SECRET_KEY",
    "af-south-1",
)

fs_copy_dir(
    "/",
    SRC_DIRECTORY,
    fs,
    TARGET_DIR,
    walker=fs_walker(
        exclude=["*.html"]
    ),
)

The first argument to fs.copy_dir is the local filesystem, given as the root of a Linux or Docker filesystem in this case.

If Will McGugan sounds familiar, he’s also the author of the more recent rich, rich-cli and textual. He’s a prolific source of brilliant Python modules!