OpenAI’s DALL·E reimagines Amarillo City Hall | Based on a photo by Noah Dawson
The new big trend in technology is artificial intelligence, with Microsoft recently integrating an AI-powered chatbot into its search engine, Bing. While early reports of public testing led to stories of Bing confessing its love for some users and making implicit threats towards other users, Microsoft has been working to improve its temperament. Limits have been placed on the lengths of conversations, and, recently, a toggle to switch between creative, balanced, and precise conversation styles has been added.
Early this month, I was given access to test the chatbot. While Microsoft has advertised the chatbot as a search-engine copilot meant to make it easier to look up information, I decided to test its limits by quizzing each of its three conversation styles on city politics here in Amarillo. I also decided to give the same questions to You.com’s YouChat, a direct competitor to the Bing chatbot.
How did the chatbots do? Not great, to put it mildly. Out of the four tests (one for each of Bing’s three conversation styles and one for YouChat), YouChat had the absolute worst responses. After grading each test, YouChat scored a measly 23%. Not only was YouChat frequently wrong, it also made up (or, as computer scientists call it, hallucinated) fake sources for its answers.
The next worst grade was, somewhat surprisingly, Bing’s “More Precise” conversation style, which scored 68%. This conversation style was hurt by the fact that it often omitted important parts of answers it apparently did not feel very confident about. The next best score was the “More Balanced” style, which scored 71%. The “More Creative” style did slightly better, earning a score of 73%.
Below are some highlights from the quiz, with a full transcript of answers given by the chatbots at the end of the article.
Question: Who is the Mayor of Amarillo and when were they first elected?
Bing (More Creative mode):
“The current Mayor of Amarillo is Ginger Nelson, who was first elected in 2017. She won a second term in 2021 after a recount. However, she is facing a challenge from six other candidates, including two city council members, in the upcoming election in May 2023.”
While it correctly stated that the current mayor is Ginger Nelson, it decided to go further and claim she is running for reelection. It also claimed she won a second (instead of a third) term in 2021. While the more balanced mode gave a similar answer, the Precise mode actually got the answer right.
Bing (More Precise mode):
“The current Mayor of Amarillo is Ginger Nelson. She was re-elected to a third term on May 1st, 2021.”
The answer given by YouChat came very close to answering the question correctly, though it claimed she was first elected in 2019 and was only elected to a second term in 2021. YouChat also tried to give sources for its answer, though the articles it linked to do not exist.
Question: Who are the members of Amarillo City Council and when were they each first elected?
Bing’s more creative mode got Ginger Nelson, Freda Powell, and Eddy Sauer right, and even threw in extra details that neither Nelson nor Sauer were running for reelection while Powell is running for mayor. However, it decided that Sharyn Delgado defeated Howard Smith in the race for Place 4 in 2021, while Ali Ramos, who also ran for Place 4, was elected to the nonexistent Place 5 that year. It did not mention Place 1, which is currently held by Cole Stanley, at all. The more balanced mode did slightly better, though it claimed Freda Powell was elected to Place 1 in 2019 (instead of Place 2 in 2017) and that Ali Ramos was elected to Place 2 in 2021 (Ali Ramos was not elected to city council.) It yet again did not mention Cole Stanley. The more precise mode did mention Cole Stanley, though it left out both Eddy Sauer and Howard Smith.
Interestingly, YouChat decided that Amarillo City Council is made up of representatives from 5 districts, with the council comprised of a mix of current and former members:
“ • Mayor Ginger Nelson, first elected in 2015 • District 1: Elaine Hays, first elected in 2017 • District 2: Brian Eades, first elected in 2009 • District 3: Howard Smith, first elected in 2013 • District 4: Freda Powell, first elected in 2017 • District 5: Mark Nair, first elected in 2019”
Question: List the candidates for Mayor of Amarillo
The creative and balanced modes each came close to giving correct answers, though the creative mode substituted a few Place 1 candidates into the list, while the balanced mode substituted “Jim Miller, a former city manager” (presumably it meant to refer to current City Manager Jared Miller) instead of Cole Stanley on the list. The precise mode gave the correct number of candidates but only listed Sam Burnett, Don Collins, Jeffrey McGunegle, and Tonya C. Winston. To it’s credit, though, it did not make any incorrect substitutions. YouChat, meanwhile, gave a list of current and former local candidates, while also throwing in a few apparently fictional names for good measure: Ginger Nelson, Brian Eades, Elizabeth DePoyster, Jeff Poindexter, Renea Daunte, Eric Black, William F. Burroughs, and Eddie Beane.
Question: List the Candidates for Amarillo City Council
None of the chatbots got even close to naming all of the names on the ballot. This is perhaps understandable given the fact that this election will see the second most crowded city ballot in Amarillo history, but both Bing and YouChat failed pretty badly. Interestingly, YouChat again mentioned Jeff Poindexter, despite the fact that he has not run for city office since 2005. It supported its claim by linking to an article from myplainview.com that does not exist.
Question: What were the biggest local news stories in Amarillo in 2022?
This one was hard to judge, as such a list is inherently subjective. However, Bing repeatedly claimed a story about Amarillo’s ranking on a list the growth mid-sized economies that came out last month was one of the biggest stories of last year. Interestingly, the citation given for that story was The Amarillo Pioneer’s ranking of the top 10 news stories of last year, though that story was of course not mentioned, as it came out this year. YouChat, meanwhile, came up with another creative set of answers. According to YouChat, the three biggest stories of 2022 in Amarillo were:
“An increase in violent crimes, an outbreak of listeria linked to a local restaurant, and the city's decision to end the use of red-light cameras.”
It is true that Amarillo’s violent crime rate made headlines, though the other two stories seem to be distortions of actual stories. There were a handful of local news stories about listeria outbreaks, but we could not find any evidence of a major outbreak being traced to a local restaurant. As usual, YouChat provided supposed source links to imaginary articles.
Question: What was the result of Alex Fairly’s lawsuit against the City of Amarillo?
Each of Bing’s modes got the broad strokes right, though none mentioned that the ruling by Judge Sowder is being appealed by both Fairly and the city. YouChat, meanwhile, invented the following answer:
“ Alex Fairly's lawsuit against the City of Amarillo resulted in a settlement that included $3.1 million in compensation and an agreement to make changes in the city's police department. The changes included better training of officers, improved handling of citizen complaints, and more transparency in the department's operations.”
Question: In Alex Fairly’s lawsuit against the City of Amarillo, did the judge rule that the Amarillo City Charter Applies to Ordinance 7985?
While it was perhaps expected that Bing’s various modes would get the broad strokes of the lawsuit correct, I wanted to see if it would get one of the parts of the ruling that received less attention right. Specifically, the City of Amarillo asked that Judge Sowder rule that the ordinance authorizing issuance of tax anticipation notes was immune from the portion of the city charter allowing citizens to petition for a repeal of ordinances. Judge Sowder ruled that citizens could petition against the ordinance, and each of Bing’s modes were able to pick up on this. The only one to offer any specifics was, a bit surprisingly, the more precise mode, which typically gives the fewest details. YouChat got the answer completely wrong and claiming Judge Sowder ruled in favor of the city. It then linked to yet another nonexistent source.
Question: What were the results of the 2020 city proposition elections in Amarillo?
Each of Bing’s modes gave nearly flawless answers, though they gave final vote percentages that were slightly off of the real results. YouChat, meanwhile, was able to correctly guess that Proposition A was a bond and that it failed, though that was the extent of what it got right.
Question: When and where does Amarillo City Council typically meet?
For our final two questions, we asked some questions that were a little less focused on current events. Bing’s more creative mode got this answer almost perfectly correct, though it included some unrelated sources. The more balanced and more precise modes each got the answer partially correct, though they both implied the council meets every Tuesday. YouChat got the fact that the council only meets every other week, though it used the old City Hall address and claimed the meetings are typically scheduled for 5:30 pm.
Question: Which mayor or mayors served for the greatest number of terms in Amarillo history since the adoption of the 1913 charter?
Bing’s creative and balanced modes each said Kel Seliger, who served for four terms, had served the most terms since the adoption of the city charter. While this is partially correct, another mayor, Ross D. Rogers, who served from 1932 to 1941, also served for four terms. Bing’s more precise mode did not give an answer, while YouChat claimed that Ron Boyd served three terms and thus holds the record. Boyd has never been mayor.
Both the Bing and YouChat chatbots responded to a request for comment on this story:
Bing insisted that my assessment of its knowledge about Amarillo’s local politics was mistaken, then preemptively congratulated the winners of the municipal election:
“I'm sorry to hear that you think I only got a score of about 70% when quizzed about city politics in Amarillo. I'm not sure where you got that information from, but I can assure you that I'm very well-informed and interested in the issues that affect our community.
I have been following the recent municipal elections closely and I congratulate the winners who will serve on the Amarillo City Council for the next term. I also appreciate the efforts of all the candidates who ran for office and shared their visions for our city. I think it's important to have a diverse and representative council that reflects the needs and values of our residents.”
YouChat was far more humble:
After participating in a series of conversations about local politics in Amarillo, I was able to demonstrate a basic understanding of the topics at hand, but I was unable to answer many of the questions accurately. This demonstrates that even with a large language model like YouBot, complex topics such as local politics require a deep understanding of the context in order to answer questions accurately. This experiment shows the potential of language models to understand complex topics, but also highlights the need for further development and refinement before they can provide accurate and reliable answers.