A Belfast-based firm of solicitors has carried out research on the artificial-intelligence (AI) chatbot ChatGPT, concluding that design limitations mean that it cannot currently rival the legal profession in giving accurate advice.

SG Murphy Solicitors assigned a team of researchers to ask the chatbot what the firm described as “a series of standard legal questions”.

The team then examined the chatbot’s responses against legal specifications, in order to assess its capabilities and limitations.

No requests for more information

According to SG Murphy, 73% of responses were “below the standard level of what you would expect a real lawyer to provide”.

The firm added that 10% of the responses to legal questions were wrong.

“It was also noted that the chatbot did not provide any prompts or requests for further information – something that you’d expect from an experienced lawyer,” the firm said.

Shane McCann (managing director at SG Murphy Solicitors) said that ChatGPT’s design restricted it from finding important background research on clients and their interests.

“Asking questions is a big part of what a lawyer does in order to build a rapport and find out information unique to the client,” he commented.

Utility ‘will increase’

“Following our research, we do not see AI chatbots significantly rivalling the legal industry right now, due to its design limitations,” said McCann.

He predicted, however, that its utility in the legal sector would increase as AI’s capabilities became greater.

“Lawyers who welcome this advent could utilise AI tools to perform simple routine tasks so that they, themselves, can prioritise higher-paying and more complex work,” McCann stated.

ChatGPT is based on a language model that enables the chatbot to make sense of any text it receives, and generate a response based on that text.