I've realized that I kinda miss the student era. It was strangely fun to handle yourself while being broke,
often you'll have to find brilliant solutions to complete your tasks, being them passing an exam without cheating,
get someone to offer you a beer before the concert begins, or just survive starvation with what you got in
the fridge and your top chef's cooking skills.
This project kinda reminds me of those times, even though we can agree that there are many solutions that are available from many providers for such a service such as CloudFlare Images or Imgix, the journey of creating a new one from scratch still offers much more enjoyment to my mind. If the game is too easy, is also tasteless.
What Is This All About?
The main purpose of the Serverless Image-Service is to manage the images that belong to a website. For today's standards, it's usual that images you see on any website are stored somewhere in a cloud datacenter from where they're requested once a page within a Web App is loaded, almost always with local and edge Caching solutions to avoid stress and improve performance for commonly accessed assets.
Serving images however might hide a drawback when considering the wide range of devices and connection quality that are available all around the world, in Romania we're pretty much covered since we're in the top 5 fastest internet speeds in the world, but this might not be the case for more remote countries and regions. That's why most of the time it would be useful to make some edits such as downscaling and lower quality for heavy images to adapt both the device resolution and the internet connection to provide a smooth and pleasant experience for any type of a user that is using our Web App.
This service allows not only to serve images over a custom domain but it also provides endpoints to upload and delete those images, that can be invoked manually or more commonly through a CMS solution.
By taking advantage of modern web services, in this case, provided by AWS, we can accomplish such a solution, guaranteeing stable functionality while also decreasing costs and allowing customization, qualities that other service providers of premade solutions of this kind cannot.
Custom Domain
Even though the service is creating a CloudFront Distribution to handle calls from API Gateway and cache
results of the GET endpoint to avoid useless common Lambda triggers, instead of just using the randomhash1234.cloudfront.net domain you might
want to proxy traffic directly from any of your domains. Sometimes people reserve a subdomain such as cdn.domain.com or
images.domain.com to serve this purpose along with their production Web App that runs on the plain
domain domain.com but most of the time, for logistic and security reasons, big tech companies will use a completely separate domain for
serving assets, this is up to your choice, remember that it might come in handy when debugging and
deploying, since the domains are different, the chance of messing up and confusing the Web App and the Image Service
while performing DNS or cache operations is lower.
This service allows you to set up almost all resources needed for hosting over your custom domain beside the
ACM Certificate, it's just enough for you to pay attention to the Setup Steps and configure the settings.yml with your
domain of choice, you can find more info about it in the Docs
Custom CDN
As a suggestion, the strategy I recommend is not to leave CloudFront alone to do the caching and distributing role, but instead, if you can afford it, put in front of it another Custom CDN service that might have more access points around the world, if you're planning to get traffic from many geographical locations. I've been working with CloudFlare to serve this purpose and it's doing great, besides the 250+ datacenters also offer an advanced caching solution for free over each one of them!
This makes the service have a double cache to ensure images are served as quick as possible from anywhere in the world! Images are static assets, and unless you're going through some heavy redesign, they won't change in time so often. Just look at any Ecommerce website, once they upload images for a specific product, those images will more likely be the same for years to come, so they're fine if cached everywhere, you won't need to worry about them any longer.
The only drawback of this approach is that you'll have to pay attention while debugging your service, and
remember to purge or invalidate the cache from both CloudFront and your Custom CDN to reach back to the origin in case
the image you're requesting has been cached in the past.
Ephemeral Storage
I was not sure how to call this paragraph but might not be related to what you're thinking. There are mainly 2 strategies when it comes to content storing and serving static assets, it's all about speed:
- An original image once uploaded is already transformed within all its possible or most used variants, dimensions, quality, watermarks... and each variant is stored as well within a persistent storage so that can be retrieved the fastest way possible without needing processing.
- Only the original image is stored and processed when requested. This of course will increase loading time,
something that nobody wants, but at the same time, with the above
double caching solution, it will end up in the so called ephemeral storage, the cache is not persistent, will invalidate itself after a while, max-age for custom CDNs usually being around a month.
Almost everyone is using the second option because it presents more advantages even if speed may look slower from its description:
- The Cache storage, even being non-persistent, allows for better spread across the world and more specifically from data centers to end clients.
- The fact that just original assets are stored means that there is a decrease in costs due to less storage and egress data from the persistent storage, assets are extracted and processed only once or a couple of times before they get stored in the cache.
- If an asset or many need to be changed, it's enough an invalidation of the cache to make the request reach the origin for a single specific file, process it back with edits, and cache it back. In a persistent situation of each variant, this will make for more work due to having to reprocess the asset and replace it within the storage, making it also potentially unavailable to the end clients during the process.
Serverless Is The New Way
I have a love/hate relationship with Serverless, it's undeniable that it does its job well, but sometimes it's a real pain in the ass to work with. I can't explain this with words, the best way to understand it is to try it. Even though it has some flaws it's undeniable that it's a powerful tool that can help you get your resource deployed within many Cloud providers with just a couple of lines of code. This of course involves lots of trial and error to make it work but the result it's a simple solution and alternative to writing for example raw CloudFormation Configs.
This service is inspired by another similar solution called Serverless Sharp. I decided to extend this service's functionalities because it looked not maintained anymore even though it had much potential. I saw that the author is still pushing over a different branch some redesign but in my opinion, it's making the whole solution overcomplicated by using OOP and schemas to map query parameters. It's a valid solution but again, way much complicated for its basic purpose.
Many things have been changed from that, you can get a better understanding by reading the GitHub README
Processing Images With Sharp
As stated above, the most interesting feature of the service is the ability to process and transform images on the fly whenever requested. This is possible with the implementation of the Sharp library. The way this works is by appending query parameters to the GET request of any file for the image to be first processed and returned as soon as possible. The list of supported options mapping can be found in the README
domain.com
Resize width to 500px
Setup And Deploy
Please read the Setup Steps carefully. Even though Serverless is a powerful tool, there are still a couple of things that you need to handle yourself such as the ACM Certificate creation and/or the commenting of settings to avoid unwanted configs to be applied when deploying.
Debugging
Once deployed, you can take advantage of the serverless offline module to mock Lambda to your localhost
and receive all logs in your terminal when invoking any endpoint.
There are Postman and Thunder Client Collections and Environments inside the GitHub Repo, that you can import into your tool of choice to
have a quick start for making endpoint requests.
Features
Below there is a breakdown of all the service's features
Upload Assets 📖
This is the way you'll want to use for uploading images from your custom CMS to the S3 Bucket so that
they can then be fetched from your Web App.
Successfull POST Request from Thunder Client over the Upload Endpoint
It's a POST endpoint that allows a multipart/form-data Content-Type body. The idea is to send the
raw binary data representing the images you want to upload and let Lambda take care of everything else.
Files will upload into the S3 Bucket you deployed, with the corresponding MIME Type and Metadata
over the requested subfolder or path mostly known as key within AWS.
If just one file with the same name under the requested subpath exists will throw an error with details.
Remove Assets 📖
Use this to remove images you uploaded into the S3 Bucket.
Successfull DELETE Request from Thunder Client over the Remove Endpoint
It's a DELETE endpoint that allows an application/json Content-Type body. It works just like the
above upload endpoint, removing the file names you specified in the files key within the request body
from the AWS subpath you made the request over.
If just one file with the requested name under the requested subpath doesn't exist will throw an error
with details.
List Assets 📖
Currently used for debugging purposes only, this endpoint lists all the AWS keys from the root
path that is currently available, aka. all the images that are uploaded in the S3 Bucket at the moment
of the request.
Successfull GET Request from Thunder Client over the List Endpoint
It's a basic GET endpoint that can be invoked on the root path of your custom domain.
Serve Assets 📖
This is the most important endpoint from the service, meant to be consumed by Web Apps and return either the original image if no processing is requested (no query parameter passed) or transforms and modifies the original image before returning it to the client.
Successfull GET Request from Thunder Client over the Get Original Endpoint
It's a GET endpoint that needs to be invoked over the full path of the filename you request within
the S3 Bucket with additional query parameters to describe eventual transforms.
Successfull GET Request from Thunder Client over the Get Processed Endpoint
Notice the
sizedifference from the original and the processed image.
Integration Ideas
Having a custom Image Service can be a smart move to boost the productivity across all the services that might consume it. Since it's independent of any CMS or Client App it means that can be easily integrated with both of them, including mobile apps. Thanks to cache it allows on the fly edits for single or all assets:
Did you just change the company logo but images have the old watermark?
Just update the new watermark in the /assets/ folder, deploy and purge the assets you want to invalidate from the cache, BOOM, now images have the new logo without any service interruption!
You need to load assets with different resolutions for different device types?
Just implement on both the Web and Mobile Apps on the front-end level some logic to request images with a different w=
query parameter, BOOM, now both your apps will reach the bliss point of quality and performance on any device type!
Would you like to use it as backup storage for assets that you upload from a CMS?
Just introduce logic to push the files also to the POST endpoint and upload them to the S3 Bucket from your
CMS, then add logic for the GET endpoint as a fallback in case your custom solution is already running in production.
What Now?
This project was a rushed one, took me about a month and a half to get it to this point. I'll be taking care of it as much as I can, you can check out the TODO if you want to see what will probably be implemented next, until then I'll go get a beer...
Cheers! 🍻
