Write an FAQ entry for why LLMs are not suitable for this bot

[dharmab]

Points to cover:

• Air combat communication is surprisingly easy to parse because it turns out it is a context-free grammar
  • Doesn't even require recursive descent - can be parsed accurately in a simple LTR order
• GCI controller logic is mostly really easy to implement, barely an inconvenience
  • Most of the decision tree follows strict criteria that can be expressed in a few hundred lines of code per workflow, tops
  • 90% of the problem space is middle school geometry
  • 10% of the problem space is college level statistics
  • LLMs are bad at math, traditional code is great at math
• The areas where humans accel - understanding the tactical and strategic context of the airspace - requires both significant familiarity, training and practice as well as a briefing of the mission plan, success and fail criteria. This is not easy for mission creators/server operators to prompt to an LLM
• Hand-written code can be unit-tested once and then be trusted to continue working. AI models must be re-validated with each change, using novel data that does not match the training data. This is labor intensive work
• Customizing hand-written code is as easy as opening a text editor and typing into it. Customizing an LLM requires a training data set. Collecting this training data has a myriad of privacy and ethical issues, especially in regions with good protections like the EU. Alternatively, a synthetic training data set presents an alternate set of issues around data bias (in the statistical sense).
• Traditional code is highly predictable, which is good for the developer and user
• LLMs are a poor choice for low-latency systems, while hand-written code runs in literally under a millisecond
• Requirement to run on a local machine without cloud APIs prevents use of best-in-class LLMS
• LLMs will bloat the system requirements, which are already strained by the other 3 AI models this software uses
• Most LLMs require either C++ or Python which is painful to test and debug in a Go project