Contractors beware: Do not trust AI to interpret regulations

Given the advancements of the past year, you might be wondering: Can federal contractors trust AI to interpret regulations?

We tested it. You be the judge.

AI, analyze a FAR part 22 clause for us

We asked ChatGPT to “Briefly summary the requirements of FAR 22.1003-4(d)(1)”

It responded:

FAR 22.1003-4(d)(1) pertains to labor standards for contracts exceeding $2,500. This subsection requires contractors to submit weekly certified payroll records, ensuring compliance with prevailing wage rates and fringe benefits as stipulated in the contract.

ChatGPT

Not quite. What (d)(1) really does is list contract types that are exceptions to the Service Contract Act and Fair Labor Standards Act. Yes, it does pertain to labor standards, but the clause in question is really about when they don’t apply, not when they do, as ChatGPT made it seem.

So, can federal contractors trust AI to interpret regulations? So far, not great.

AI, analyze a DFARS clause for us

We then asked “What does DFARS 252.225-7057 require”

It responded:

DFARS 252.225-7057 is a clause related to the acquisition of certain foreign products. It typically addresses restrictions or requirements for the use of certain minerals, like tungsten and tantalum, to ensure compliance with regulations related to conflict minerals. Always refer to the latest version of the DFARS or consult legal experts for the most up-to-date information.

ChatGPT

Swing and a miss.

DFARS 252.225-7057 requires the contractor to disclose to the government when it employs or intents to employ individuals who will perform work on the contract within the People’s Republic of China.

ChatGPT’s answer is totally wrong. In fact, we’re not sure what clause it’s summarizing.

So, can federal contractors trust AI to interpret regulations? Zero for two.

OK, that one is pretty obscure though. What about the limitation on subcontracting? Most people know that one.

AI, what does the FAR say about the Limitation on Subcontracting?

We asked “What does FAR 52.219-14 require?” It said:

FAR 52.219-14 is a clause that requires a contractor to limit subcontracting to small businesses. Specifically, it mandates that the prime contractor must perform at least 50% of the cost of the contract incurred for personnel with its own employees.

ChatGPT

Overall, not a bad answer. It is true that the FAR clause requires the prime contractor to perform at least 50 percent of the contract, but that’s not the whole story—and it’s phrased as though that’s all there is to it. The 50 percent requirement applies only to supplies or services contracts. There are different limitations for general construction (85 percent) and special trade construction (75 percent) contracts. ChatGPT made no mention of them.

If a contractor had relied on that answer, it could have been way off. Lawyers are well aware that there are some serious ethical issues with using AI in the practice of law. The cautionary tales are already starting to pile up.

But the question again is, can federal contractors trust AI to interpret regulations? Contractors can choose their own path, of course. There’s no doubt that AI is a powerful tool.

But if this quick test is any indication, at this point federal contractors would be wise not to trust AI to interpret regulations.