Welcome to The Blueprint, a new feature where we highlight how Google Cloud customers are tackling unique and common challenges across industries using the latest AI and cloud technologies. We hope to inspire others looking to innovate in their work.
Founded in 2018, Voicify reimagines the traditional phone call with the goal of transforming every call into a seamless and engaging experience.
The challenge:
When we started Voicify in 2018, our vision was to help organizations build confident, pragmatic, and technically grounded voice-driven assistants for any channel, including phones and chat. But the pandemic changed everything.
We shifted our focus to telephone use cases primarily in the restaurant and healthcare sectors where, at the time, call volume and staffing posed significant challenges. Restaurants could potentially miss up to 20% of their calls and lose orders as a result, and healthcare providers struggled to keep up with call volume with the required 100% accuracy when integrating appointment information into a practice management system. We realized that specialized, purpose-driven AI assistants were the key to businesses maintaining excellent service at scale.
To succeed, we had to overcome four primary challenges:
-
Transactional precision: Our voice assistant needed to reason with complex customer requests against point of sale and practice management systems with 100% accuracy.
-
Traffic spike management: Our LLM usage needs to be provisioned accurately to keep costs down and maintain customer services in spite of the common (and extreme) spikes in traffic seen in restaurants and healthcare organizations.
-
Latency: Any delay in the assistant’s response can cause customers to hang up. We needed superfast time to first token, with minimal delay from when a user sends a voice or text request to when the AI model generates its first piece of output.
-
Security and compliance: Since our founding in 2018, we’ve ensured that we’re HIPAA, SOC2, ISO27001, and PCI-compliant, and that our security is enterprise-grade. We needed architecture and infrastructure that employs all possible safeguards to safeguard data integrity and security.
The solution:
Our conversational orchestration platform builds and validates restaurant orders against a point-of-sale system before submission to ensure accuracy. Under the hood, Gemini Flash, served via Gemini Enterprise Agent Platform, vastly improves latency, minimizing user wait times and preventing hang-ups. With it, we also see approximately 25% to 30% savings compared to our previous use of other LLMs, and with greater reliability too.
To grow the business — and call volume — and to handle traffic spikes, we switched from Google AI Studio to Vertex AI and its current incarnation in Gemini Enterprise. We wanted the enterprise guarantees the latter provided, which we needed for scaling as well as for security and compliance for our healthcare clients. Specific Gemini Enterprise Agent Platform features help us manage high call volumes without experiencing service interruption or dropped responses.
These enterprise-grade services may have carried an increased cost over AI Studio, but they were well worth it to ensure reliable uptime, and the premium pay-as-you-go feature made scaling much easier for us. For example, we used a combination of provisioned throughput and premium pay-as-you-go with Vertex AI to accommodate all-time high usage the day before Thanksgiving, and we saw no rate limiting issues.
The architecture:
The outcome:
Gemini has reduced the burden on our in-house programmatic tools for pulling context and building menus. We’ve seen great improvements in performance and reliability, with lower latency and greater reliability with Gemini. And, the increased stability of our Gemini-powered assistants has made client onboarding much more efficient. Now it only takes one to two days to get a restaurant ready to test after gaining access to the POS system, down from what previously took one to two weeks.
With Google solutions for scale and enterprise-grade service, we’ve optimized our critical time-to-first-token metric, minimizing customer wait times. Using Vertex AI’s provisioned throughput and pay-as-you-go features, we’ve ensured 100% uptime, prevented dropped responses and rate-limiting issues, even during periods of all-time high usage. We’re now able to easily manage the spiky nature of restaurant traffic.
In terms of technology, we anticipate moving beyond conversational order capture to more proactive assistance, using context from conversations or POS activities. Your typical Friday night order from your favorite Japanese restaurant? Someday soon it might be Voicify’s voice assistant proactively placing it for you.
The details:
Our industry focus presents a few unique challenges that we had to spend time solving within the backend.
The core component of our Voicify solutions is our voice orchestration platform, which manages the entire phone AI stack and is designed for enterprise-grade scalability and security. This is also the node where industry solutions are called depending on user needs.Our voice orchestration platform sits close to the customer and coordinates backend services like Gemini and the different components of the voice assistant. We use it to manage functions like automated speech recognition, text-to-speech, and text generation, which is not purely generative but includes programmatic elements.
One of the unique architectural decisions we made was how to manage large, complex restaurant menus. We decided to avoid putting the entire menu into a single prompt, and we include only certain information in the initial prompt and then gather more details as the conversation progresses. This improves response times and helps manage the complexity of larger orders by focusing on only the relevant parts of each menu in a given interaction.
We also designed the architecture from the outset of our company to meet the high standards of enterprise clients for security and compliance, particularly in healthcare. We are making sure that our scalability is enterprise-grade. Architecturally we’re also employing all safeguards to ensure data integrity and safety too. Lastly, our platform is designed to support a multicloud environment as part of our strategy for achieving the highest possible level of availability.