Legacy Modernization meets GenAI

GenAI era: A timeline of key events

One primary reason for the
current wave of hype and excitement around GenAI is the
versatility and high performance of general-purpose LLMs. Each new generation of these models has consistently
shown improvements in natural language comprehension, inference, and response
quality. We are seeing a number of organizations leveraging these powerful
models to meet their specific needs. Additionally, the introduction of
multimodal AIs, such as text-to-image generative models like DALL-E, along
with AI models capable of video and audio comprehension and generation,
has further expanded the applicability of GenAIs. Moreover, the
latest AI models can retrieve new information from real-time sources,
beyond what is included in their training datasets, further broadening
their scope and utility.

Since then, we have observed the emergence of new software products designed
with GenAI at their core. In other cases, existing products have become
GenAI-enabled by incorporating new features previously unavailable. These
products typically utilize general purpose LLMs, but these soon hit limitations when their use case goes beyond
prompting the LLM to generate responses purely based on the data it has been trained with (text-to-text
transformations). For instance, if your use case requires an LLM to understand and
access your organization’s data, the most economically viable solution often
involves implementing a Retrieval-Augmented Generation (RAG) approach.
Alternatively, or in combination with RAG, fine-tuning a general-purpose model might be appropriate,
especially if you need the model to handle complex rules in a specialized
domain, or if regulatory requirements necessitate precise control over the
model’s outputs.

The widespread emergence of GenAI-powered products can be partly
attributed to the availability of numerous tools and development
frameworks. These tools have democratized GenAI, providing abstractions
over the complexities of LLM-powered workflows and enabling teams to run
quick experiments in sandbox environments without requiring AI technical
expertise. However, caution must be exercised in these relatively early
days to not fall into traps of convenience with frameworks to which
Thoughtworks’ recent technology radar
attests.

Problems that make modernization expensive

When we began exploring the use of “GenAI for Modernization”, we
focused on problems that we knew we would face again and again – problems
we knew were the ones causing modernization to be time or cost
prohibitive.

• How can we understand the existing implementation details of a system?
• How can we understand its design?
• How can we gather knowledge about it without having a human expert available
  to guide us?
• Can we help with idiomatic translation of code at scale to our desired tech
  stack? How?
• How can we minimize risks from modernization by improving and adding
  automated tests as a safety net?
• Can we extract from the codebase the domains, subdomains, and
  capabilities?
• How can we provide better safety nets so that differences in behavior
  between old systems and new systems are clear and intentional? How do we enable
  cut-overs to be as headache free as possible?

Not all of these questions may be relevant in every modernization
effort. We have deliberately channeled our problems from the most
challenging modernization scenarios: Mainframes. These are some of the
most significant legacy systems we encounter, both in terms of size and
complexity. If we can solve these questions in this scenario, then there
will certainly be fruit born for other technology stacks.

The Architecture of CodeConcise

CodeConcise is inspired by the Code-as-data
concept, where code is
treated and analyzed in ways traditionally reserved for data. This means
we are not treating code just as text, but through employing language
specific parsers, we can extract its intrinsic structure, and map the
relationships between entities in the code. This is done by parsing the
code into a forest of Abstract Syntax Trees (ASTs), which are then
stored in a graph database.

Figure 2: An ingestion pipeline in CodeConcise.

Edges between nodes are then established, for example an edge might be saying
“the code in this node transfers control to the code in that node”. This process
does not only allow us to understand how one file in the codebase might relate
to another, but we also extract at a much granular level, for example, which
conditional branch of the code in one file transfers control to code in the
other file. The ability to traverse the codebase at such a level of granularity
is particularly important as it reduces noise (i.e. unnecessary code) from the
context provided to LLMs, especially relevant for files that do not contain
highly cohesive code. Primarily, there are two benefits we observe from this
noise reduction. First, the LLM is more likely to stay focussed on the prompt.
Second, we use the limited space in the context window in an efficient way so we
can fit more information into one single prompt. Effectively, this allows the
LLM to analyze code in a way that is not limited by how the code is organized in
the first place by developers. We refer to this deterministic process as the ingestion pipeline.

Figure 3: A simplified representation of how a knowledge graph might look like for a Java codebase.

Subsequently, a comprehension pipeline traverses the graph using multiple
algorithms, such as Depth-first Search with
backtracking in post-order
traversal, to enrich the graph with LLM-generated explanations at various depths
(e.g. methods, classes, packages). While some approaches at this stage are
common across legacy tech stacks, we have also engineered prompts in our
comprehension pipeline tailored to specific languages or frameworks. As we began
using CodeConcise with real, production client code, we recognised the need to
keep the comprehension pipeline extensible. This ensures we can extract the
knowledge most valuable to our users, considering their specific domain context.
For example, at one client, we discovered that a query to a specific database
table implemented in code would be better understood by Business Analysts if
described using our client’s business terminology. This is particularly relevant
when there is not a Ubiquitous
Language shared between
technical and business teams. While the (enriched) knowledge graph is the main
product of the comprehension pipeline, it is not the only valuable one. Some
enrichments produced during the pipeline, such as automatically generated
documentation about the system, are valuable on their own. When provided
directly to users, these enrichments can complement or fill gaps in existing
systems documentation, if one exists.

Figure 4: A comprehension pipeline in CodeConcise.

Neo4j, our graph database of choice, holds the (enriched) Knowledge Graph.
This DBMS features vector search capabilities, enabling us to integrate the
Knowledge Graph into the frontend application implementing RAG. This approach
provides the LLM with a much richer context by leveraging the graph’s structure,
allowing it to traverse neighboring nodes and access LLM-generated explanations
at various levels of abstraction. In other words, the retrieval component of RAG
pulls nodes relevant to the user’s prompt, while the LLM further traverses the
graph to gather more information from their neighboring nodes. For instance,
when looking for information relevant to a query about “how does authorization
work when viewing card details?” the index may only provide back results that
explicitly deal with validating user roles, and the direct code that does so.
However, with both behavioral and structural edges in the graph, we can also
include relevant information in called methods, the surrounding package of code,
and in the data structures that have been passed into the code when providing
context to the LLM, thus provoking a better answer. The following is an example
of an enriched knowledge graph for AWS Card
Demo,
where blue and green nodes are the outputs of the enrichments executed in the
comprehension pipeline.

Figure 5: An (enriched) knowledge graph for AWS Card Demo.

The relevance of the context provided by further traversing the graph
ultimately depends on the criteria used to construct and enrich the graph in the
first place. There is no one-size-fits-all solution for this; it will depend on
the specific context, the insights one aims to extract from their code, and,
ultimately, on the principles and approaches that the development teams followed
when constructing the solution’s codebase. For instance, heavy use of
inheritance structures might require more emphasis on INHERITS_FROM edges vs
COMPOSED_OF edges in a codebase that favors composition.

For further details on the CodeConcise solution model, and insights into the
progressive learning we had through the 3 iterations of the accelerator, we
will soon be publishing another article: Code comprehension experiments with
LLMs.

In the subsequent sections, we delve deeper into specific modernization
challenges that, if solved using GenAI, could significantly impact the cost,
value, and time for modernization – factors that often discourage us from making
the decision to modernize now. In some cases, we have begun exploring internally
how GenAI might address challenges we have not yet had the opportunity to
experiment with alongside our clients. Where this is the case, our writing is
more speculative, and we have highlighted these instances accordingly.

Reverse engineering: drawing out low-level requirements

When undertaking a legacy modernization journey and following a path
like Rewrite or Replace, we have learned that, in order to draw a
comprehensive list of requirements for our target system, we need to
examine the source code of the legacy system and perform reverse
engineering. These will guide your forward engineering teams. Not all
these requirements will necessarily be incorporated into the target
system, especially for systems developed over many years, some of which
may no longer be relevant in today’s business and market context.
However, it is crucial to understand existing behavior to make informed
decisions about what to retain, discard, and introduce in your new
system.

The process of reverse engineering a legacy codebase can be time
consuming and requires expertise from both technical and business
people. Let us consider below some of the activities we perform to gain
a comprehensive low-level understanding of the requirements, including
how GenAI can help enhance the process.

Thoughtworks worked with the client mentioned earlier to explore ways to
accelerate the reverse engineering of a large legacy codebase written in COBOL/
IDMS. To achieve this, we extended CodeConcise to support the client’s tech
stack and developed a proof of concept (PoC) utilizing the accelerator in the
manner described above. Before the PoC, reverse engineering 10,000 lines of code
typically took 6 weeks (2 FTEs working for 4 weeks, plus wait time and an SME
review). At the end of the PoC, we estimated that our solution could reduce this
by two-thirds, from 6 weeks to 2 weeks for a module. This translates to a
potential saving of 240 FTE years for the entire mainframe modernization
program.

High-level, abstract explanation of a system

We have experienced that LLMs can help us understand low-level
requirements more quickly. The next question is whether they can also
help us with high-level requirements. At this level, there is much
information to take in and it is tough to digest it all. To tackle this,
we create mental models which serve as abstractions that provide a
conceptual, manageable, and comprehensible view of the applications we
are looking into. Usually, these models exist only in people’s heads.
Our approach involves working closely with experts, both technical and
business focussed, early on in the project. We hold workshops, such as
Event
Storming
from Domain-driven Design, to extract SMEs’ mental models and store them
on digital boards for visibility, continuous evolution, and
collaboration. These models contain a domain language understood by both
business and technical people, fostering a shared understanding of a
complex domain among all team members. At a higher level of abstraction,
these models may also describe integrations with external systems, which
can be either internal or external to the organization.

It is becoming evident that access to, and availability of SMEs is
essential for understanding complex legacy systems at an abstract level
in a cost-effective manner. Many of the constraints previously
highlighted are therefore applicable to this modernization
challenge.

In the era of GenAI, specifically in the modernization space, we are
seeing good outputs from LLMs when they are prompted to explain a small
subset of legacy code. Now, we want to explore whether LLMs can be as
useful in explaining a system at a higher level of abstraction.

Our accelerator, CodeConcise, builds upon Code as Data techniques by
utilizing the graph representation of a legacy system codebase to
generate LLM-generated explanations of code and concepts at different
levels of abstraction:

• Graph traversal strategy: We leverage the entire codebase’s
  representation as a graph and use traversal algorithms to enrich the graph with
  LLM-generated explanations at various depths.
• Contextual knowledge: Beyond processing the code and storing it in the
  graph, we are exploring ways to process any available system documentation, as
  it often provides valuable insights into business terminology, processes, and
  rules, assuming it is of good quality. By connecting this contextual
  documentation to code nodes on the graph, our hypothesis is we can enhance
  further the context available to LLMs during both upfront code explanation and
  when retrieving information in response to user queries.

Ultimately, the goal is to enhance CodeConcise’s understanding of the
code with more abstract concepts, enabling its chatbot interface to
answer questions that typically require an SME, keeping in mind that
such questions might not be directly answerable by examining the code
alone.

At Thoughtworks, we are observing positive outcomes in both
traversing the graph and generating LLM explanations at various levels
of code abstraction. We have analyzed an open-source COBOL repository,
AWS Card
Demo,
and successfully asked high-level questions such as detailing the system
features and user interactions. In this occasion, the codebase included
documentation, which provided additional contextual knowledge for the
LLM. This enabled the LLM to generate higher-quality answers to our
questions. Additionally, our GenAI-powered team assistant, Haiven, has
demonstrated at multiple clients how contextual information about a
system can enable an LLM to provide answers tailored to
the specific client context.

Discovering a capability map of a system

One of the first things we do when beginning a modernization journey
is catalog existing technology, processes, and the people who support
them. Within this process, we also define the scope of what will be
modernized. By assembling and agreeing on these elements, we can build a
strong business case for the change, develop the technology and business
roadmaps, and consider the organizational implications.
Without having this at hand, there is no way to determine what needs
to be included, what the plan to achieve is, the incremental steps to
take, and when we are done.

Before GenAI, our teams have been using a number of
techniques to build this understanding, when it is not already present.
These techniques range from Event Storming and Process Mapping through
to “following the data” through the system, and even targeted code
reviews for particularly complex subdomains. By combining these
approaches, we can assemble a capability map of our clients’
landscapes.
While this may appear like a large amount of manual effort, these can
be some of the most valuable activities as it not only builds a plan for
the future delivery, but the thinking and collaboration that goes into
making it ensures alignment of the involved stakeholders, especially
around what is going to be included or excluded from the modernization
scope. Also, we have learnt that capability maps are invaluable when we
take a capability-driven approach to modernization. This helps modernize
the legacy system incrementally by gradually delivering capabilities in
the target system, in addition to designing an architecture where
concerns are cleanly separated.

GenAI changes this picture a lot.

One of the most powerful capabilities that GenAI brings is
the ability to summarize large volumes of text and other media. We can
use this capability across existing documentation that may be present
regarding technology or processes to extract out, if not the end
knowledge, then at least a starting point for further conversations.
There are a number of techniques that are being actively developed and
released in this area. In particular, we believe that
GraphRAG which was recently
released by Microsoft could be used to extract a level of knowledge from
these documents through Graph Algorithm analysis of the body of
text.
We have also been trialing GenAI over the top of the knowledge graph
that we build out of the legacy code as mentioned earlier by asking what
key capabilities modules have and then clustering and abstracting these
through hierarchical summarization. This then serves as a map of
capabilities, expressed succinctly at both a very high level and a
detailed level, where each capability is linked to the source code
modules where it is implemented. This is then used to scope and plan for
the modernization in a faster manner. The following is an example of a
capability map for a system, including the source code modules (small
gray nodes) they are implemented in.

Nevertheless, we have learnt not to view this entirely LLM-generated
capability map as mutually exclusive from the traditional methods of
creating capability maps described earlier. These traditional approaches
are valuable not only for aligning stakeholders on the scope of
modernization, but also because, when a capability already exists, it
can be used to cluster the source code based on the capabilities
implemented. This approach produces capability maps that resonate better
with SMEs by using the organization’s Ubiquitous language. Additionally,
comparing both capability maps might be a valuable exercise, surely one
we look forward to experimenting with, as each might offer insights the
other does not.

Discovering unused / dead / duplicate code

Another part of gathering information for your modernization efforts
is understanding within your scope of work, “what is still being used at
all”, or “where have we got multiple instances of the same
capability”.

Currently this can be addressed quite effectively by combining two
approaches: static and dynamic analysis. Static analysis can find unused
method calls and statements within certain scopes of interrogation, for
instance, finding unused methods in a Java class, or finding unreachable
paragraphs in COBOL. However, it is unable to determine whether whole
API endpoints or batch jobs are used or not.

This is where we use dynamic analysis which leverages system
observability and other runtime information to determine if these
functions are still in use, or can be dropped from our modernization
backlog.

When looking to find duplicate technical capabilities, static
analysis is the most commonly used tool as it can do chunk-by-chunk text
similarity checks. However, there are major shortcomings when applied to
even a modest technology estate: we can only find code similarities in
the same language.
We speculate that by leveraging the result of our capability
extraction approach, we can use these technology agnostic descriptions
of what large and small abstractions of the code are doing to perform an
estate-wide analysis of duplication, which will take our future
architecture and roadmap planning to the next level.

When it comes to unused code however, we see little or no use in
applying GenAI to the problem. Static analysis tools in the industry for
finding dead code are very mature, leverage the structured nature of
code and are already at developers’ fingertips, like IntelliJ or Sonar.
Dynamic analysis from APM tools is so powerful there’s little that tools
like GenAI can add to the extraction of information itself.

On the other hand, these two complex approaches can yield a vast
amount of data to understand, interrogate and derive insight from. This
could be one area where GenAI could provide a minor acceleration
for discovery of little used code and technology.
Similar to having GenAI refer to large reams of product documentation
or specifications, we can leverage its knowledge of the static and
dynamic tools to help us use them in the right way for instance by
suggesting potential queries that can be run over observability stacks.
NewRelic, for instance, claims to have integrated LLMs in to its solutions to
accelerate onboarding and error resolution; this could be turned to a
modernization advantage too.

Idiomatic translation of tech paradigm

Translation from one programming language to another is not something new. Most of the tools that do this have
applied static analysis techniques – using Abstract Syntax Trees (ASTs) as intermediaries.

Although these techniques and tools have existed for a long time, results are often poor when judged through
the lens of “would someone have written it like this if they had started authoring it today”.

Typically the produced code suffers from:

Assuming an organization embarks on this journey, it will soon face lengthy testing and verification
cycles to ensure the generated code behaves exactly the same way as before. This becomes even more challenging
when little to no safety net was in place initially.

Despite all the drawbacks, code conversion approaches continue to be an option that attracts some organizations
because of their allure as potentially the lowest cost/ effort solution for leapfrogging from one tech paradigm
to the other.

We have also been thinking about this and exploring how GenAI can help improve the code produced/ generated. It
can’t help all of those issues, but maybe it can help alleviate at least the first three or four of them.

From an approach perspective, we are trying to apply the principles of
Refactoring
to this – essentially
figure out a way we can safely and incrementally make the jump from one tech paradigm to another. This approach
has already seen some success – two examples that come to mind:

Conclusion

Today’s landscape has numerous opportunities to leverage GenAI to
achieve outcomes that were previously out of reach. In the software
industry, GenAI is already playing a significant role in helping people
across various roles complete their tasks more efficiently, and this
impact is expected to grow. For instance, GenAI has produced promising
results in assisting technical engineers with writing code.

Over the past decades, our industry has evolved significantly, developing patterns, best practices, and
methodologies that guide us in building modern software. However, one of the biggest challenges we now face is
updating the vast amount of code that supports key operations daily. These systems are often large and complex,
with multiple layers and patches built over time, making behavior difficult to change. Additionally, there are
often only a few experts who fully understand the intricate details of how these systems are implemented and
operate. For these reasons, we use an evolutionary approach to legacy displacement, reducing the risks involved
in modernizing these systems and producing value early. Despite this, the cost/time/value equation for
modernizing large systems is often prohibitive. In this article, we discussed ways GenAI can be harnessed to
turn this situation around. We will continue experimenting with applying GenAI to these modernization challenges
and share our insights through this article, which we will keep up to date. This will include sharing what has
worked, what we believe GenAI could potentially solve, and what, however, has not succeeded. Additionally, we
will extend our accelerator, CodeConcise, with the aim of further innovating within the modernization process to
drive greater value for our clients.

Hopefully, this article highlights the great potential of harnessing
this new technology, GenAI, to address some of the challenges posed by
legacy systems in the industry. While there is no one-size-fits-all
solution to these challenges – each context has its own unique nuances –
there are often similarities that can guide our efforts. We also hope this
article inspires others in the industry to further develop experiments
with “GenAI for Modernization” and share their insights with the broader
community.