Leveraging AI to streamline student access to information

At Four Kitchens, we’re among the leaders in considering how AI should be applied in a thoughtful and efficient way at higher ed institutions. Yale came to us looking for a way to streamline how their students could get answers from complicated data.

We worked with the university to create askYale, an AI-powered chatbot. We started by working with the school’s campus dining data, which was comparatively low-stakes from an informational standpoint. But this was Yale’s aim with the pilot program. If the results were wrong, anyone testing the chatbot would obviously recognize the issue so it could be resolved quickly.

The rollout was successful, and askYale is now expanding its use to supply information about health services and other university datasets. Here are three insights we gathered from this five-week sprint project:

From a technical standpoint, implementing an AI chatbot is not the most daunting aspect of a project like this. The real challenge lies in preparing your data to train your AI tool effectively.

Depending on your institution, you may face a unique challenge if your chatbot needs to integrate data from multiple sources. In Yale’s case, much of the chatbot’s data was being drawn from a Drupal website. But the university also used a separate application to manage dining hall menus. We ended up running a scraper over the public site to ensure askYale gathered all the data it needed.

Universities already have a wealth of content, which serves as an excellent starting point for a chatbot. The key is to organize this information into a format that AI can easily digest. This is where a retrieval-augmented generation (RAG) system comes into play.

Leveraging RAG for enhanced AI performance

Large-language models (LLMs) that power AI software generate answers to specific prompts by querying their databases, which introduces a potential for errors. RAG is an architecture that reduces the likelihood of bad results by prompting the AI tool to base its answer only on the data it finds. The model will not use outside information to fill in the gaps in its responses.

RAG results in more accurate and contextually relevant outputs. By importing your curated source data into a RAG, you create a robust yet reliable foundation for your AI chatbot to draw from when answering queries.

When it comes to developing an AI-powered tool, the expression “garbage in, garbage out” couldn’t be more apt. The quality of your chatbot’s responses is directly related to the quality of the content in its database. Well-organized, relevant, and contextually rich content is crucial for ensuring a chatbot is equipped to select the most appropriate information in its responses.

To optimize your content for AI, consider the following strategies:

• Write with clarity in mind, using simple language and straightforward explanations
• Organize information with clear headings and bulletpoints
• Be explicit in your instructions, avoiding inferred references
• Ensure your content is relevant and up-to-date

AI systems rely entirely on the information provided to them. Therefore, it’s crucial to review and refine your content regularly, ensuring it remains understandable, accurate, and comprehensive. This ongoing process of content curation and optimization is key to maintaining a high-performing AI chatbot that enhances the student experience.

Testing AI-powered chatbots requires a paradigm shift from traditional software testing methodologies. Most internet software applications up to now have been declarative, much like a jukebox: You put in your quarter, you get your song. The results are predictable and static based on your input.

With generative AI, you don’t know what song you’re going to get. The open-ended nature of AI presents a unique challenge: How do you test a system where users can ask any question and receive a wide variety of responses?

Testing an AI chatbot like askYale requires flexibility, adaptability, and a willingness to embrace uncertainty. It’s important that you assign subject matter experts who know the material your chatbot is drawing from to verify the results are meaningful and make sense.

Safeguarding your chatbot against bad actors

Along with testing against desired or predictable questions, you also must test how your chatbot will respond to queries that probably shouldn’t be asked. We added safeguards to askYale that made it harder for bad actors to interact with the chatbot to deliberately produce results that would harm the university’s reputation.

The tool will check the prompt and score it to determine whether the question merits a response — “Where can I find biscuits on campus?” — versus the kinds of queries a hospitality tool should not answer.

To further refine askYale’s results about dining on campus, we had our testers ask questions multiple times and inquire in different ways to identify patterns. If you’re getting successful answers eight, nine, or 10 times out of 10, then great. If not, then you have to adjust two variables:

• Prompt: Allows you to give your chatbot personality in its responses. We told the chatbot it was a big fan of Handsome Dan (the school’s mascot), and it loved using emojis. These elements helped give the bot something like a personality.
• Temperature: Establishes how the chatbot should behave. At one end of the temperature spectrum, the responses from your AI chat offer direct interpretations of the questions asked. At the other end, the results are more unpredictable. You ask for specific information and the bot returns a chaotic answer.

By striking a balance between both variables, you ensure your chatbot’s answers are appropriate for your user. Continuous evaluation and adaptation are key. In the process, you’ll discover new situations to test as well as potential improvements, ensuring the chatbot remains a reliable resource.

As universities navigate the AI landscape, our experience with askYale shows that implementing chatbots can significantly improve student experiences. By working with an experienced partner and maintaining proper oversight, institutions can harness AI’s potential to streamline how you communicate vital information.

The key to your success lies in prioritizing data quality and adopting thorough testing processes to maximize AI’s capabilities. By doing so, you can create AI-powered tools that not only meet your students’ immediate needs, but also adapt to the evolving demands of higher education.

As ‌technology continues to advance, universities that embrace the responsible integration of AI will be better positioned to support students, faculty, and staff. If this sounds like an application that will help your institution’s communication priorities, we should talk.

If you’re attending HighEdWeb 2024 in Albuquerque, make sure to see our askYale session live.

“Revolutionizing conversations: AI chat integration on YaleSites”
Where: Albuquerque Convention Center, Room 220–230, Albuquerque, NM
When: Monday, September 23, 2024, 11:15am to noon
More information: Registration details