Agentic Processes For Table Q&A

Can you? Yes. But, Should You?

Fully agentic text2sql is yet another flavor of the day. Today, as was the case a few years ago, the ability to instantly resolve an ask via SQL without getting locked into a dashboard is in much demand. There’s been work in the area that people have made careers around. The recent emergence of chatbots like ChatGPT has rekindled this excitement in the area. While the field is rapidly evolving and progressing, a few important items can be noteworthy to take a look at given where we stand.

Back in the middle of march 2023, a team of researchers (Liu et. al) from UIC published this paper on arXiv. It examined ChatGPT’s text2sql capabilities. Here’s Bing Chat’s summary of the paper:

FWIW, that is the abstract of the paper and the summary simply draws from it.

As you can see, the excitement in the area is definitely warranted. If you double click a little bit more, there are some results there that shows that completely agentic processes may not be the way to go today. For e.g. in their paper the researchers publish a set of benchmark metrics and you can see that execution accuracy (EX) across the board can be a hit or miss

Credits: Liu et al. A comprehensive evaluation of ChatGPT’s zero-shot Text-to-SQL capability

Instantly, we can also see that most SQL that these models generate end up being valid (VA on the table above). It doesn’t take a lot of insight to put the two together to know that making production apps that use fully agentic processes that generate SQL for execution can be, let’s say, not a bright idea at the moment. But that does that mean we should never use agentic processes? Definitely not. For what its worth, it might be a decent idea to explore Single Table Q&A this way so long as the results consumed by the user is heavily caveated. That being said, the cool kids in the block have released a bunch of open source tools to get us started without a lot of heavy-lifting. Let’s explore that for quick starting our exploration for text to results. In fact, outside the scope of the usage in IDEs which deliver intelligent auto complete suggestions, most of the folks who use sql generation actually want results from an utterance.. not just SQL. Understanding the practical needs here, let’s pivot to single table text to results. The tools that aid us here in this quick exercise are LangChain, ChromaDB, Spark and Pandas. There really isn’t a lot of explain except the high level process. You have a set of tables in a schema that need to be independently queried. How do you make this happen reasonably with LangChain?

Highlevel View of the Process

The same process can be done using PandasAI as well. The lift here is next to nothing. But let’s go through the high level execution quickly

Here are a bunch of tables that I have on Databricks SQL.I can quickly generate the metadata for each of the tables

Push them into a Chroma collection

Instantiate an LLM to be used with the LangChain Spark Dataframe Agent (notice it would send the first record of the Dataframe to the LLM). You could replace this LLM with a model from HuggingFace like Flan-T5 for example. But at the moment, OpenAI (GPT3.5) seems to be generating the best results (from my subjective experience).

Ask away!

Note: While this task shows the right result, if your prompt is ambiguous or unclear you might end up with some incorrect result. In general, this provides fairly decent single table Q&A capabilities

Interestingly, Pandas AI also provides some privacy capabilities as well. You can check it out if that is of interest to you. These capabilities can easily be made to delivered as an API via Databricks Model Serving. But the takeaway here is — currently one needs a hybrid approach to perform “text to results”. I plan to go through this on another blog at another time. But, it heavily borrows from this blog and one of my other blogs from the past.