Python download set of files via s3 keys

You can also give a name that is different from the object name. If your file is existing as a. Including the sub folders in your s3 Bucket. If you have any issues, you can also comment below to ask a question. Spread the knowledge by sharing : 0 More. Share via. Copy Link. Powered by Social Snap. Copy link. Copy Copied. Possibly duplicate- stackoverflow.

Add a comment. Active Oldest Votes. Bucket bucketName for obj in bucket. Improve this answer. Konstantinos Katsantonis Konstantinos Katsantonis 9 9 silver badges 16 16 bronze badges. But you didn't set credentials! Credentials can be set in different ways. See boto3. Show 2 more comments. HolyGuacamole 53 1 1 silver badge 5 5 bronze badges. Soulduck Soulduck 1 1 gold badge 4 4 silver badges 13 13 bronze badges.

Better to avoid putting your keys in your code file. At worst, you can put your keys in a separate protected file and import them. It's also possible to use boto3 without any credentials cached and instead use either s3fs or just rely on the config file reddit. Probably not. It should work out to about same as looping over each key with boto3 maybe with an added call to list objects, but you need that in both cases — hume.

Another approach building on the answer from bjc that leverages the built in Path library and parses the s3 uri for you: import boto3 from pathlib import Path from urllib. Matthew Cox Matthew Cox 7 7 silver badges 19 19 bronze badges. We have to filter the suffix after we have the API results, because that involves inspecting every key manually. For example:. This is the function I have in my codebase:.

This function absorbs all the messiness of dealing with the S3 API, and I can focus on actually using the keys. Posted 18 July Tagged with aws , python. Pass the continuation token into the next response, until we reach the final page when this field is missing. JPG' : print key.