The AI-Native Contact Center
Building customer experience from the ground up

The easiest part of building an LLM application is the LLM API. The hardest part is everything else…

It’s easy to become captivated by the shiny objects that promise automation and efficiency. Generative AI  and the proliferation of Large Language Models is the latest shiny object that has changed the perception of what’s possible in the enterprise. Developers and corporate executives have been led to believe that securing an LLM API from the emerging set of providers is all that’s needed to deliver enterprise-wide transformation. “If all you have is a hammer, everything looks like a nail” — something that is clearly visible with Generative AI right now. Beyond the model, you’ll also have to think about everything else that’s required to deliver a successful production ready solution. The “Everything else” is what’s required to build an LLM application;  namely (a) accessing the right data, (b) designing inputs/prompts, (c) fine-tuning for a specific task, (d) delivering outputs safely and securely.

Introducing Avaamo LLaMB™

As mentioned above, everything else turns out to be harder than you might think and that’s why we built Avaamo LLaMB™.  LLaMB™ is a framework that lets you build, deploy and maintain LLM applications in the enterprise.

What is the Everything Else?

Lets delve deeper into the “everything else “ that goes into enterprise deployments and how LLaMB™ has been built to solve these challenges:

1. Trust and security:

Raw Model APIs when piped directly into the enterprise are very vulnerable. Moreover, these models are subject to adversarial attacks such as prompt injection. By carefully or maliciously crafting the prompts, one can “lure” the models to give unintended responses and influence their behaviors. “The everything else” is of major import here and is critical to enterprise security teams.

How LLaMB™ protects data and delivers trust

•  We use a standard protocol to encrypt the prompt when we send it to an LLM. We also mask the personally identifiable information (PII) in prompts we send. Data masking is an extra measure of protection that replaces PII in a prompt with placeholder data, keeping your sensitive data safely stored inside your source data repository.

•. Zero retention of your data: We have zero retention agreements in place with LLM providers. These agreements mean you and your users don’t need to worry about how external LLM providers retain your data. The model forgets the prompt and the response as soon as the response is sent back to the user.

•. When a response is generated, we scan it for toxicity and log it into our audit trail before we return it to the user.

Exhitbit 1

How LLaMB delivers, safely and securely

2. Domain fine-tuning your enterprise use case 

LLMs are initially trained on extensive generic datasets. Consequently, they offer limited utility for enterprises until they undergo fine-tuning to comprehend your specific dataset and the domain relevant to your use case. This fine-tuning process involves acquiring an understanding of the terminology used in domains such as HR, IT, customer service, or any other domain you intend to apply the use case within.

How LLaMB™ solves this

This is why LLaMB™comes with a pre-tuned layer containing thousands of datasets related to support tickets, HR requests, and customer service questions. It offers a ready-to-use toolkit for initiating the development of an LLM application in these specific domains.

Exhitbit 2

Fine-tuning an LLM to comprehend your dataset and the domain relevant to your use case

3. Prompt design 

LLMs have weird APIs. You provide a natural language input in the form of a “prompt” and get a probabilistic response back. It turns out that mastering this API requires a lot of tinkering and experimentation: to solve a new task, you’ll probably need to try a lot of different prompts (or chains of prompts) to get the answer you’re looking for. 

Becoming comfortable with the probabilistic nature of LLM output is a process that requires time and thorough testing to grasp the boundaries and nuances of your prompts. It involves choosing the right words, phrases, symbols, and formats that guide the model in producing high-quality and relevant text. This endeavor demands both time and substantial trial and error, not to mention valuable computing resources. Unfortunately, the various developer tools available to streamline this iterative process are complex and time intensive . Writing Python strings declared as constants in code is not for the faint-hearted.

How LLaMB™ solves this?

 LLaMB™ makes it easy by providing a Prompt Builder with 1) ready to use prompt library of templates for several domains and a 2) prompt designer that does not require knowledge of any coding language. It enables enterprise developers to build prompts quickly and effectively.

4. Data integration

LLM applications achieve their highest potential when they can access data tailored to their specific tasks, allowing them to generate or synthesize information within that context. This necessity implies that applications require access to private or proprietary data systems, such as HR policy documents or support knowledge bases.

Building a useful LLM application involves providing the language model with relevant “context.” This context is particularly crucial to prevent LLMs from generating inaccurate information or “hallucinating”. By supplying proper context, the model can extract accurate data from documents rather than inventing it. Essentially, this process equates to giving LLMs a form of “memory,” a feature not inherently present in current foundational models.

While obtaining the right data is pivotal, it’s equally important to manage the credentials and access controls for multiple internal systems with caution. Applications must diligently monitor users’ access privileges to prevent unintended disclosure of private or sensitive information to the model or other users. Unfortunately, due to the lack of proper infrastructure, many folks are currently reinventing the wheel in this regard.

This complexity extends far beyond a proof of concept on a personal laptop with a few documents. Enterprises must contend with the management of hundreds of thousands of pieces of content in a global setting, which may be overseen individually or by groups of content editors located in various global locations, updating information asynchronously from time to time.

How  LLaMB™ solves this:

LLaMB™ has developed an advanced data ingestion and content filtering layer. This layer offers seamless out-of-the-box integration with enterprise systems and allows for the monitoring and support of existing enterprise access privileges.

Additionally, LLaMB™ has introduced real-time content syncing, ensuring that content can remain in its existing repositories. This innovation significantly reduces the typically burdensome and intricate process associated with content migration.

Furthermore, LLaMB™ has gone even further by providing automated parsing of complex tables and structures. This simplifies the ingestion of various dimensions of enterprise content, essential for enabling business users to obtain meaningful answers to their inquiries.

5. Deployment

The most frequently mentioned issue when utilizing LLM APIs directly is “hallucination.” Ensuring that the LLM behaves as intended poses a significant challenge. Hallucination, a prevalent failure mode, occurs when the model generates responses that sound plausible but are ultimately incorrect or nonsensical. If your prompt lacks sufficient context, the model may provide a generic or inaccurate response.

How LLaMB™ uses dynamic grounding for deployment:

When a user submits a request, we translate that request into a prompt. A prompt serves as our means of instructing an LLM regarding the specific task we want it to perform. It comprises task instructions and can incorporate relevant task-specific data and contextual information. This process of enhancing a prompt with data is referred to as “dynamic grounding.” The decision of whether to include data in a prompt, what type of data to include, and how much to include depends on the particular use case.

LLaMB™ ensures trust throughout the user experience by attributing the source, often multiple sources, from the enterprise corpus to provide the user with the context of the answer.

In general, a more grounded prompt tends to yield a more valuable response. Furthermore, LLaMB™ ensures trust throughout the user experience by attributing the source, often multiple sources, from the enterprise corpus to provide the user with the “context of the answer.”

Budgeting and costs

Model APIs and cloud GPUs are expensive. As of this writing, renting a single machine with 8 A100s on AWS costs over $23k per month if left running. Hastily written code (e.g., infinite loops, bad API calls) or poor infrastructure management can quickly run up huge bills. To develop applications responsibly, you need to be able to manage the code you’re running, its model usage, and the expenses associated with it.

LLaMB's business model focuses on generation rather than tokens.

How LLaMB™ optimizes costs

LLaMB uses various techniques for token optimization and to reduce the cost of ownership (which is passed on directly to the customer).

•. By storing frequently accessed data, you can improve response times without needing to make repeated calls to our API.

•. Out-of-the-box ability to use cached data for repetitive queries whenever possible and invalidate the cache when new information is added.

•. A business model that focuses on generation rather than tokens.

Summary

As laid out above, building LLM-powered applications isn’t just about the model. Enterprises should think through the “Everything else” very carefully before embarking on a generative AI project. As with all software, you’ll have to think about data, cloud infrastructure, credentials, security, latency, tools and cost, and the rapidly evolving nature of LLM applications considerably amplifies those challenges.

Our goal with LLaMB™ is to improve the experience and ergonomics for building LLM applications by providing an effective toolkit for the “everything else” beyond the LLM API.  LLaMB™ makes it easy to build, deploy and maintain LLM applications safely and efficiently.

Author(s)

Ram Menon is the CEO and co-founder of Avaamo.