Managing AWS Lambda Versions with AWS Step Functions: A comprehensive guide

Managing AWS Lambda Versions with AWS Step Functions: A comprehensive guide

[English version only] 

Each AWS account has a regional default storage limit for Lambda and Lambda Layers (Zip archives) of 75 GB. This sounds a lot, and it can even be increased by demand, but many functions combined with frequent deployments might lead to storage pollution quite quickly. Therefore, it makes sense to clean up old deployment packages regularly. This blogpost introduces an AWS Step Functions state machine that is designed to automate the identification and deletion of older Lambda function versions.

Starting point

AWS creates a new version for each lambda function deployment and keeps them all available – just in case an older one is needed in future. Every deployment package is stored and counts to the overall size limit of 75 GB. Even though this amount of storage is quite large it is advisable (and recommended by AWS) to clean up older versions which are no longer needed on a regular basis.

As of now, AWS does not provide a direct solution to manage these versions, prompting developers to create custom solutions. One example of a custom solution is the use of a Lambda function like the Lambda Janitor by Yan Cui. Our idea for an implementation was a state machine – an automated and visual solution for the Lambda version management.

This state machine handles all Lambda functions (optionally targeting specific ones based on predefined criteria) and deletes / removes old versions based on threshold which determines how many versions should be kept.
A visual representation of the final workflow as well as a brief explanation of the most important steps is given below:

Retrieving and filtering the function names

The workflow begins with calling and listing the names of all existing Lambda functions within the AWS region. The “lambda:listFunctions” API call returns up to 50 items per call and a marker token in case more results are available. To take this into account the paginator pattern after the “Iterate Functions” map state is being used.

For the first step we transform the result with the ResultSelector to extract only the function names. By doing that we get an output like this:

Ein Bild, das Text, Screenshot, Schrift, Reihe enthält. Automatisch generierte Beschreibung

To iterate through the array of lambda function names in the list, we use a Map state, which allows us to set a concurrency limit of 5. This limit helps to control the number of simultaneous requests and prevents throttle calls.

In our case we just want to clean up versions of Lambda functions which belong to our project. Others which are managed by the platform team need to stay untouched.

The Lambda functions from our project have a specific prefix in the name, that makes it easy for us to filter through all functions in the region. To do that, we added a “Choice” step to determine whether the state machine should continue the version cleanup or ignore the current Lambda function. This prefix can also be retrieved from the state machine invocation event.

Ein Bild, das Text, Screenshot, Schrift, Zahl enthält. Automatisch generierte Beschreibung

Getting all the available versions

In the following step, all available versions are retrieved using the “lambda:listVersionsByFunction” API call. The outcome of this step is shown below – a list of all versions and the name of the function currently processed.

Ein Bild, das Text, Screenshot, Zahl enthält. Automatisch generierte Beschreibung

Keeping the function name in this step and adding it to the ResultPath is mandatory because we will need it again in a few steps when deleting the versions.

The “lambda:listVersionsByFunction” API call returns up to 50 items and a marker token to retrieve more results. To take care of the pagination – the paginator pattern would have been required again. However, we have decided not to implement it to keep the workflow simpler.

There are not that many deployments in our environment that we would reach more than 50 versions per Lambda function before the workflow runs the next time. In addition, the workflow which is triggered by an AWS EventBridge scheduler could run more often to avoid this situation – for instance, every day instead of every week – just as an example.

Removing the $LATEST version and get version count

The Lambda API returns the versions in a way that the latest (called $LATEST 😊) one comes first followed by all other versions in ascending order – from oldest to newest. We decided not to rely on this implicit order but to exclude the $LATEST version explicitly from all further processing steps as well as to sort the remaining version numbers – just in case AWS is going to change the response format.

Unfortunately, AWS Step Functions does only provide a limited set of array and JSON processing functions – so called intrinsic functions. A Lambda function is required to perform the necessary work. Hopefully, AWS adds some more power to the intrinsic function’s palette so that these kind of simple helper functions will no longer be necessary in future.

The outcome of this step consists of the lambda function name, the array of versions to be processed and their count.

Check number of available versions and remove outdated ones

In the next step, the state machine compares the number of available versions against a threshold. This is done with a “Choice” State. In our case we wanted to always keep the three most recent versions + the $LATEST one, so the “version_count” is compared with 3:

If the number of available versions exceeds the threshold we set, the oldest version (first position in the array) is deleted using the “lambda: deleteFunction” API call in the step “Delete oldest version”.

After that, a modification of the payload is required as the deleted version number and the version count itself needs to be adapted. A “Pass” state is used to remove the oldest version (the first position in the array with the versions) and to decrease the version counter.

Ein Bild, das Text, Screenshot, Software, Zahl enthält. Automatisch generierte Beschreibung

The execution keeps checking and deleting old versions until their number has reached the threshold value (three in this example).

Ein Bild, das Text, Screenshot, Diagramm, Reihe enthält. Automatisch generierte Beschreibung

 

One remaining thing to consider related to Lambda Aliases

A Lambda alias reference one or two function versions which cannot be deleted if the alias exists. It is possible to retrieve all aliases of a function via the “lambda:listAliases” API call. One way to make sure to take aliases into account would be to get all involved versions and to remove them from the versions array before the version deletion action. This requires some custom code in a Lambda function.

Another option – which we have implemented – consists in defining an error handler for the “Delete old version” step. This one captures the “Lambda.ResourceConflictException” which is thrown when trying to delete a version which cannot be deleted for instance because of an alias reference. This error handling makes sure that the state machine does not fail in case of alias references or other unforeseen problems when trying to perform the deletion.

A screenshot of a computer screen Description automatically generated

The state machine finishes after all Lambda functions which have been supposed to be processed do not possess more than the most recent versions – all others have been deleted. This mechanism helps to prevent deployment package pollution when used regularly.

Conclusion

In this guide on managing AWS Lambda versions with AWS Step Functions, I’ve shown how developers can leverage automation to streamline the management of Lambda function versions. By automating these processes, teams can focus more on development rather than maintenance, maintaining operational efficiency and resource optimization. Overall, this solution provides a practical, visual tool for managing Lambda functions effectively, helping users navigate the complexities of cloud operations with greater ease.

A new way of collaboration: A 3-Day PI Planning in VR using Arthur Software

A new way of collaboration: A 3-Day PI Planning in VR using Arthur Software

[English version only] 

In the ever-evolving landscape of software development, effective planning and collaboration have become essential for success. However, gathering large teams of 40 or more people in a physical space for a Program Increment (PI) Planning event can be logistically challenging and costly. Fortunately, advancements in technology have brought about exciting opportunities, and through the use of virtual reality (VR) and e.g. Arthur Software, teams can now experience a transformative planning process without the limitations of physical space.

In this blog post, we explore the benefits of conducting a 3-day PI Planning event in VR using Arthur as we did in one of our last projects. Such an event is of course preceded by planning. Here, too, we would like to offer you a brief insight. At the end of this blog post, you will receive an overview of the benefits of PI Planning using Virtual Reality as well as an overview of what you should consider in advance when planning a (PI Planning) workshop in VR.

Day 0: Preparation

One of the keys to successful PI planning in VR, is preparation and coordination with all parties involved. This includes classic activities such as thematic discussions and coordination, but also requires knowledge in the field of working in virtual reality. For an optimal transfer of an onsite workshop to an event in the virtual world, VR experts should be involved at an early stage.

Day 1: Setting the stage for collaboration

The first day of the VR-based PI Planning event focuses on building a strong foundation for collaboration among the participants. With the help of Arthur, teams are immersed in a virtual environment that fosters a sense of presence and engagement. The day begins with a system demo via MS Teams, where teams can showcase their work, providing valuable insights and updates to the larger group. These demos facilitate cross-team learning and create an environment of shared knowledge. Since these demos are a presentation without any interaction, we decided to use MS Teams as the presentation platform. Working in VR can be tough for some people in the beginning, so these sessions should have a dedicated benefit e.g., regarding the interaction during the meeting.

After the demos we switched to VR and the teams engage in retrospectives, reflecting on the previous increment and identifying areas for improvement. In the virtual space, teams can utilize virtual whiteboards and collaboration tools provided by Arthur to document their insights, fostering transparency and shared understanding. It quickly became clear that having an external person take over the moderation and documentation was a great advantage, as it allowed the participants to concentrate fully on the content of the event.

The day continues with discussions around the product vision and goals. Using Arthur’s powerful visualizations, teams can explore the product roadmap, key metrics, and customer needs. Through interactive presentations and discussions, participants gain a comprehensive understanding of the strategic objectives and align themselves towards a shared vision.

Day 2: Planning the next PI

On the second day, the focus shifts to the detailed planning. Participants gather in virtual breakout sessions, divided into smaller groups to work on specific features, epics, or user stories. With Arthur’s VR environment, teams can easily collaborate and brainstorm ideas, sketch out workflows, and define acceptance criteria on virtual whiteboards. In contrast to MS Teams breakout sessions, the participants in VR can easily switch from on group to another to have a little chat or discuss open points. VR offers a wide range of implementation options. It was therefore necessary to weigh up the options together with the VR experts in advance in order to select the most suitable ones. As this was many people’s first experience of working in VR, it was particularly important that an expert was available at all times to answer questions and provide suggestions.

During this day, the teams also receive valuable insights into the business context surrounding the project. For this session a switch to Microsoft Teams was useful to give the participants a little break from the VR experience. This knowledge empowers teams to make informed decisions during the planning process, ensuring alignment between the business objectives and technical implementation.

At the end of the day, the teams present their roadmaps to the larger group. Using Arthur’s virtual presentation capabilities, participants can showcase their plans, highlight dependencies, and clarify their intentions. This fosters transparency and encourages cross-team collaboration by identifying potential synergies and opportunities for shared resources. Due to the interactive possibilities in Arthur or in VR in general, the development teams gain a big benefit in comparison to collaboration platforms like MS teams. As you know from the »face-to-face meetings« it is a lot easier to interact, discuss and decide things when not only one person is able to speak at the same time.

Requirements elicitation and prioritization in VR

Day 3: Building a cohesive plan and reflection

The final day of the PI Planning event focuses on building an overall plan and addressing dependencies and risks. Teams gather virtually to discuss and align their individual roadmaps, ensuring cohesion and synchronicity across the project. Arthur’s VR environment enables participants to easily visualize and understand the interdependencies between different teams and features. Once again, it became clear that bringing together the individual contents is always a challenge, regardless of the location, which makes customized support in this step all the more important.

The event concludes with a PI retrospective, where teams reflect on the planning process, capture lessons learned, and identify areas for improvement. Especially several feedback for the planning itself and the new experience using VR were very helpful for improving future plannings. It was exiting so see how much people were happy and motivated to use a new technology and how this improves the team performance in several ways. Of course, there is room for improvement, like every new method or technology every team must evaluate how this can be used in the most efficient way. Most of the mentioned points regarding improvement are related to the fact that the majority of participants were using Virtual Reality for the first time. Topics like get to know the hard- and software are crucial for a successful planning.

Project plans and dependencies can be discussed even more intensively within the team thanks to the interactivity in VR

Conclusion

Conducting a 3-day PI Planning event in virtual reality using Arthur software offers numerous benefits for large teams. By embracing VR technology, teams can overcome the logistical challenges of physical gatherings and unlock new levels of collaboration and efficiency by reducing travel costs. With immersive experiences, powerful visualizations, and interactive tools, VR enables teams to align their efforts, plan effectively, and address potential risks. As the world continues to embrace remote work and virtual collaboration, leveraging VR technology in PI Planning becomes an exciting opportunity for organizations to stay ahead in the fast-paced world of software development.

PI Planning in VR – key points at a glance

Main benefits

  • Travel cost savings
  • Better time management through elimination of travel time
  • Competitive advantage for organizations to stay ahead in the fast-moving business world
  • Increased efficiency and better collaboration through immersive experiences, powerful visualizations, and interactive tools
  • Strengthened team spirit through collaboration in VR
  • Improved team performance through interactive involvement of all team members

What needs to be considered in advance?

  • Concept development incl. preparation and coordination of all persons involved in close cooperation with VR experts
  • Selection of workshop design and methodology in VR
  • Technical onboarding of the participants
  • Moderation and documentation, ideally by an external person, so that all participants can focus entirely on the execution and added value in VR
  • Support and advice for workshop participants when it comes to realizing their ideas and wishes

 

Contact us at any time if you have questions about this or if you are planning a PI Planning event or a similar event in VR and wish to receive support. We will be happy to accompany you through these steps and beyond!

 

These topics might be of interest to you

Webinar | Smart durch den Berufsalltag: Künstliche Intelligenz im Job | 1. Februar 2024 | 10:00 – 11:00 Uhr

Workshop | Lerne deinen neuen KI Kollegen kennen | 22. Februar 2024 | 9:00 – 16:30 Uhr

Website | Business XR (Extended Reality)

Website | Artificial Intelligence for Your Digital Transformation

Blog | Data Visualization in Virtual Reality