Uploading to S3 using signed URLs

There are a few guides to doing this out there - however, I found that none of them really worked for me for one reason or another. So I thought I’d document what worked for me. If nothing else it’s an alternate path to the one many blog posts take.

In this scenario I have an Express application serving APIs on the back end; the Express app is where the S3 signed URLs get generated. But to step back a bit...

What are signed URLs?

There’s nothing magic about these. Say you have three entities - the web client, the app server, and a storage server. Your web app needs to upload some files to the storage server.

You don’t want to route all uploads to the storage server via your app server, for obvious reasons. You also don’t want to share the keys used to access the storage server with the whole world, so you can’t just embed them in the webpage.

Instead the client sends a request to the app server, essentially saying “I have this file I would like to upload”. The server has access to the keys. If it decides the request is okay then it takes that information, builds a URL with it which can be used for PUT or POST, and then signs it, appending the signature to the URL. The signature can be checked by the entity receiving the PUT/POST, whatever it is.

The URL to be signed can include as much or as little information as you want - file type, file size, file name, whatever. It can be set to expire at a fixed time, too, so you don’t end up opening a backdoor by leaking URLs.

LocalStack

I’ve been using LocalStack for testing. I didn’t want to keep uploading to S3 the whole time:

• I want to be able to work without a network connection
• partly because it’s faster to work locally
• partly because I needed something easy to control for testing
• for cost reasons.

LocalStack is easy to set up and run; I’m only using S3 but you have the AWS suite available to you in mock-up form. There is a paid version if you need more of the features. The free (open source) version works for S3 and most things, though.

aws4

Initially I started by using the Amazon AWS SDK. Seems reasonable, right?

Unfortunately, whatever I did I couldn’t get the thing to generate a URL that LocalStack (or Minio, or S3 itself...) would accept. It kept on returning 503 permission denied errors. From what I read it’s likely to be a HTTP header which is different between signature and request, but I tried aws4 before I started debugging that in depth.

aws4 takes a simpler approach. You give it information such as which HTTP method you want to use (GET, PUT, POST, etc), the service you want to sign for, the path (or S3 key) you want to use to access the data. It returns a signed URL that you can use, and if necessary some headers to use. Easy.

axios

This is the last bit. I tried requests, as that seems to be popular. I just didn’t get on with it, though. It’s almost certainly me not them...

Anyway, axios is simple to use. If you’re going to make multiple calls then you can create an instance which you pre-configure with the common components (base URL, any custom headers, etc). Then making the call is as simple as:

axios.get(URL)
axios.put(URL, data)

Each call can take an optional config object, which is where you’d put the headers.

Example

Here’s some of the code I put together. LOCAL_AWS_HOST  is an environment variable I’ve got set to indicate when I’m using LocalStack as opposed to “real” S3.

let opts = {
	method: "PUT",
	service: "s3",
	region: process.env.AWS\_REGION,
	path: `/${S3_BUCKET_NAME}/${s3Key}`,
	signQuery: true,
};
if (process.env.LOCAL\_AWS\_HOST) {
	opts.host = process.env.LOCAL\_AWS\_HOST;
}
let signature = aws4.sign(opts);

That’s it. signature now contains the data that you passed in, plus path which is the signed URL to use, plus a header object. (The header object may be empty, but it will be there, which simplifies your logic later.)

The above is wrapped in a function (called here getSignedUrl), and used as below. I’ve omitted error handling for conciseness.

let data = {
	fileName: this.fileToUpload.name,
	fileType: this.fileToUpload.type
};
let signature = await getSignedUrl(data);
try {
	let url = "https://" + signature.host + signature.path;
	let res = await axios.put(url,
		fs.FileReadStream(fileName),
		{ headers: signature.headers }
	);
	if (res.status != 200) {
		console.error(“Error putting file”, res);
	} else {
		console.log(“Happy ending”);
	}
}

And that's all there is to it. I realise the FileReadStream is Node.js-specific - you'll have to arrange your own alternative for the browser. ;-)

That's whole lot easier than the messing around I was trying with the Amazon SDK!