Custom Layers In AWS Lambda
The logic is clear, the requirements are clear and you are really geared up to write your code in Lambda for some automation. You proceed writing your code. Pretty confident it would work but when you try to test it. You realise you cannot move forward because of a certain library not properly imported in your lambda. You go through couple of online sources to see how to package and build your own Lambda layer. Seems like a piece of cake!
You add this custom layer on to your Lambda.However, still running it causes one or another error to appear. You want to focus more on your code rather than worrying about these layers. So what is the fastest and most reliable way to do it?
Let’s see and understand in more details as to what is going on!
AWS Lambda is a computing service provided by AWS that is event driven and serverless. It runs your code without you to worry about various computing resources. All you have to do is the create Lambda function, chose an IAM role , attach policies/permissions to the role and let it know what runtimes you would like to choose e.g Python 3.8. So far so good.
Layers are zip files that contains additional codes, libraries,configuration files and other dependencies. Once you have created the Lambda function. You might need to add such layers on to your function. There are layers already available in Lambda .You can directly add them on to your function. Example — The Scipy/Numpy Layer.
For other you need to create your own custom layer. This article is about these custom layers.Certain steps are required to create them. The first is the preperation of the zip file. You need to install Docker, if you have not already .
Docker is a container that include code and dependencies so that you can run the code reliably from one computing enviornment to other.Containers allow packaging of an application with all of the parts it needs, such as libraries and other dependencies, and deploy it as one package.
Now that we have docker installed, let’s prepare our zip file.
- Start by creating an empty folder on your local computer. E.g “mylayer”
- Inside this folder create a “requirements.txt” file. Write the name of the library or dependency that you would like to use. Example-Here I would like to install “requests” . You can add more than one libraries that you want to install, if required.
3. On your terminal ,check your present working directory by typing “$pwd”. change the directory using command “ % cd yourpath/mylayer”
4. Run the following to install the libraries-
docker run -v “$PWD”:/var/task “lambci/lambda:build-python3.8” /bin/sh -c “pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit”
Note-Remember to include the runtime same as the one that your Lambda has. If you have the python 3.7 configured as runtime for Lambda. Then replace “python/lib/python3.8/site-packages/” to 3.7 in step 4.
5. Zip the content by running the following command.”zip -r requests.zip python > /dev/null” . Sometimes you may face problem with zipping this way. For that you can manually go to the folder that you created and zip the “python” file that appears after you have ran the Step 4.
This is how you get your zip file ready. Now you need to go to the layers in AWS Lambda. Click on create layers icon and name the layer. Next step involves uploading of prepared zip file. If the zip file you created is less than 10 mb, you can directly upload it by selecting “upload a .zip file” option. Else you will have to upload your zip file to s3 and paste the link to it on the “upload a file from s3 “ option.
Once the layer is created. Go to your Lambda and scroll down to “Layers”. In the layers you will find the option “add a new layer”. Click on it and just add the Custom layer that you prepared.
Having to spend less and less time on these issues always help us to be more focused on development of the logics and algorithms. I came across this particular post that helped me a lot to overcome the issues I was previously getting while creating these zip files. Hope my post be as helpful to you as this was for me.
References
