When you are hosting an Elixir/Phoenix app in DigitalOcean you may want to use Spaces, the DigitalOcean’s cloud storage alternative to AWS S3.
When you are hosting an Elixir/Phoenix app in DigitalOcean you may want to use Spaces, the DigitalOcean’s cloud storage alternative to AWS S3 (If you want to use AWS S3 instead, you can read AWS S3 in Elixir with ExAws).
Hopefully, Spaces is compatible with S3 APIs, which means that we can apply most of the things we saw in the previous article.
DigitalOcean Spaces and API Keys
Let’s start by setting app Spaces and API keys. Once logged into your DigitalOcean account, go to the spaces page and simply create a new space.
Remember that when creating a new space you’ll start paying 5$/month fee, getting a 250GB of storage and 1TB of outbound transfer. Important: once passed the bundled tier you’ll face extra charges, so please take a look at the pricing page before starting to use the service.
Once created a new Space, first we need to select the datacenter region, in this case I’ve selected New York.
As long as the communication is between a droplet and Spaces endpoint that are in the same region you will not be billed for the outbound bandwidth from Spaces to the droplet
https://www.digitalocean.com/community/questions/spaces-bandwidth-pricing
Then we choose the unique name, exactly as we did before, for the AWS S3 configuration. In this case the Space name is poeticoding-elixir-test.
Once the space is created, we are redirected to the Space page, where we see the Space url: https://poeticoding-elixir-test.nyc3.digitalocean.com.
Now, we just need to generate the Access Keys in the API page, by clicking Generate new Key in the Spaces Access Keys section.
Two keys will be generated: an Access Id key and a Secret Access Key – just copy them and keep them secret because whoever has them can access to your Spaces.
ExAws configuration
Then we add the dependencies in mix.exs
# mix.exs
def deps do
[
{:ex_aws, "~> 2.1"},
{:ex_aws_s3, "~> 2.0"},
{:hackney, "~> 1.15"},
{:sweet_xml, "~> 0.6"},
{:jason, "~> 1.1"},
]
end
and configure ex_aws and ex_aws_s3 in config/config.exs
# config/config.exs
import Config
config :ex_aws,
debug_requests: true,
json_codec: Jason,
access_key_id: {:system, "SPACES_ACCESS_KEY_ID"},
secret_access_key: {:system, "SPACES_SECRET_ACCESS_KEY"}
config :ex_aws, :s3,
scheme: "https://",
host: "nyc3.digitaloceanspaces.com",
region: "nyc3"
In the first config we configure :ex_aws, by setting the access_key_id and secret_access_key. In this case we use SPACES_ACCESS_KEY_ID and SPACES_SECRET_ACCESS_KEY environment variables to store the Spaces keys we’ve generated before.
Then we configure the S3 API endpoint with the region"nyc3" in my case ) and the host, which is "{region}.digitaloceanspaces.com".
put, list, get, delete
Similarly to the S3 case, we are now going to upload, download and delete a small file in our Space.
iex> local_image = File.read!("elixir_logo.png")
<<137, 80, 78, 71, 13, 10, 26, 10, 0, 0, ...>>
iex> ExAws.S3.put_object("poeticoding-elixir-test", "images/elixir_logo.png", local_image) \
...> |> ExAws.request!()
%{
body: "",
headers: [
{"ETag", "\"c00d85d9dfb797854a6e5db5b7a2286d\""},
{"x-amz-request-id", "tx-...-nyc3b"},
{"Date", "Tue, 03 Dec 2019 16:56:27 GMT"}
],
status_code: 200
}
To put and object we use ExAws.S3.put_object/4"poeticoding-elixir-test"), the second argument is the object key (the path) and the third argument is the file’s content.
Refreshing the Space page on the browser, now we should see the object uploaded with key images/elixir_logo.png.
We can list the objects in this Space with ExAws.S3.list_objects/2, passing the Space name as first argument.
iex> ExAws.S3.list_objects("poeticoding-elixir-test") \
...> |> ExAws.request!() \
...> |> get_in([:body, :contents])
[
%{
e_tag: "\"c00d85d9dfb797854a6e5db5b7a2286d\"",
key: "images/elixir_logo.png",
last_modified: "2019-12-03T16:56:27.000Z",
owner: %{display_name: "103381", id: "103381"},
size: "29169",
storage_class: "STANDARD"
}
]
In the response, we see the object’s key along with some other useful information like size, last modified date, and the ETag which, in Spaces, is the MD5 hash of the file.
Then, to get the object’s content we use ExAws.S3.get_object/2, passing the Space name and object’s key arguments.
iex> resp = ExAws.S3.get_object("poeticoding-elixir-test", "images/elixir_logo.png") \
...> |> ExAws.request!()
%{
body: <<137, 80, 78, 71, 13, ...>>,
headers: [
{"Content-Length", "29169"},
{"ETag", "\"c00d85d9dfb797854a6e5db5b7a2286d\""},
{"Date", "Tue, 03 Dec 2019 17:15:00 GMT"}
],
status_code: 200
}
iex> File.read!("elixir_logo.png") == resp.body
true
The request returns a response map with the whole file’s content in :body.
Then we delete the object with ExAws.S3.delete_object/2.
iex> ExAws.S3.delete_object("poeticoding-elixir-test", "images/elixir_logo.png") \
...> |> ExAws.request!()
Multipart upload and Presigned URLs
DigitalOcean Spaces supports multipart upload, so a large file can be uploaded in parts that are sent separately and in parallel to Spaces, using Elixir File.Stream without loading the whole file in memory.
In case we have a slow connection, we can pass a timeout option to ExAws.S3.upload/4. We make a multipart upload request and ExAws starts to upload the file’s parts.
If you need to fine tune the upload process, take a look at the DigitalOcean Spaces Performance Tips page, where they talk about file size and multipart uploads.
For this example I used numbers.txt, a 125MB text file that you can find in the Elixir Stream and large HTTP responses article.
iex> ExAws.S3.Upload.stream_file("numbers.txt") \
...> |> ExAws.S3.upload("poeticoding-elixir-test", "numbers.txt", [timeout: 120_000]) \
...> |> ExAws.request!()
[debug] ExAws: Request URL: "https://nyc3.digitaloceanspaces.com/poeticoding-elixir-test/numbers.txt?uploads=1" ATTEMPT 1
[debug] ExAws: Request URL: "https://nyc3.digitaloceanspaces.com/poeticoding-elixir-test/numbers.txt?partNumber=2 ATTEMPT 1
[debug] ExAws: Request URL: "https://nyc3.digitaloceanspaces.com/poeticoding-elixir-test/numbers.txt?partNumber=1 ATTEMPT 1
%{
body: "...",
headers: [ ... ],
status_code: 200
}
If we want to securely share the file making a temporarily url, we can generate presigned urls with ExAws.S3.presigned_url/5. By default the URL expires after one hour – by passing the :expires_in option we can set a different expiration time (measured in seconds).
iex> ExAws.Config.new(:s3) \
...> |> ExAws.S3.presigned_url(:get, "poeticoding-elixir-test", "numbers.txt", [expires_in: 300])
{:ok,
"https://nyc3.digitaloceanspaces.com/poeticoding-elixir-test/numbers.txt?...&X-Amz-Expires=300&...&X-Amz-Signature=..."}
We can also use a presigned URL to lazily download a file using a library like HTTPoison or Mint, and process the file’s content on the fly. More on this in Presigned URLs and Download Streams – Process a file on the fly
