Assessment of the Spider Challenge

A closer look at Machine Learning for Text-to-SQL

In an earlier article I took a critical look at the likes of the Spider Challenge, a competition to produce a machine learning apparatus to convert natural language queries into SQL.

My conclusion, on looking closely at the challenge, is that if at the time of evaluation, the ML apparatus is retrained on a new and previously unseen set of natural language queries and their corresponding SQL…then the ML apparatus is effectively given the answers. But…if there is no…