ChatGPT, LLMs, and what they mean for RSEng

[jdamerow]

ChatGPT, LLMs, and what they mean for RSEng, RSEs, etc. How are people using it, how should we be using it, how shouldn't we be using it, etc

[christophernhill]

👍

[NickleDave]

Posting some resources.

I have strong feelings about this--mainly, it's bad to replace real human thought with stochastic parrots that are fragile, not really intelligent, and also happen to destroy the environment--but I'm guessing you won't have any trouble finding pro-LLM hype articles.

CoPilot, not ChatGPT, but most relevant for RSEs, IMO

• https://matthewbutterick.com/chron/this-copilot-is-stupid-and-wants-to-kill-me.html
• https://githubcopilotinvestigation.com/
• https://githubcopilotlitigation.com/
• https://twitter.com/DocSparse/status/1581461734665367554?cxt=HHwWhMDRibPEvfIrAAAA
• nice summary blog post: https://rutgershonorsblog.wordpress.com/2023/02/15/how-the-github-copilot-lawsuit-is-going-so-far/

LLMs in science and more generally

• https://www.nature.com/articles/s42254-023-00581-4 (and see citations)
• https://irisvanrooijcogsci.com/2022/12/29/against-automated-plagiarism/
• https://irisvanrooijcogsci.com/2023/01/14/stop-feeding-the-hype-and-start-resisting/

other issues:

• LLMs suffer from model collapse: https://arxiv.org/abs/2305.17493
• LLM text is not easily detected: https://decrypt.co/149826/openai-quietly-shutters-its-ai-detection-tool
• adversarial LLM prompts are fairly easy to generate and transferable across models: https://arxiv.org/abs/2307.15043
• training LLMs consumes massive amounts of water at a time when this resource is already endangered e.g. in the southwest
  • https://arxiv.org/abs/2304.03271
  • https://techthelead.com/study-reveals-training-chatgpt-consumes-as-much-water-as-a-nuclear-reactor/

[CollinsEM]

I believe that one the most salient issues regarding the use of LLMs for professional work is the extent to which we have (or lack) a valid mental model of their competencies and a trust model for their ability to perform a given task reliably and without errors.

If you think about the recent history of humanity as increasingly competent tool users, and the development of science and engineering as a real-time virtuous design-test-evaluate evolutionary feedback loop, then one can begin to see that the most successful technological developments have been ones that are sufficiently constrained or designed in such a way that a skilled human operator can reliably operate them with high success rates. A skilled operator requires the aforementioned mental models for the embodied behavior of the technology and an understanding of (or at the very least, an intuitive feel for) its limitations.

Part of the problem with deploying these LLMs and other gen-AI tools is that they are wildly under-constrained. Their deliberately designed to give the illusion of competence, but without any of the assurances of reliability. As a practicing engineer, I would be sued into destitution if I produced a product that harmed its users in ANY significant way. Therefore, I have a very strong incentive to fully understand the technology that I am developing and the many edge cases that might arise when used or misused. These edge cases must be disclosed to the end user so that they are made aware of the limitations of the technology, can develop an appropriate trust model for its proper use, and have realistic expectations of the harms that can result form its misuse.