There’s a certain … cassandra-esque irony in people getting frustrated with LLMs not “doing what they tell them”, and are now, undestanding-by-fire why domain-specific formal languages were developed in the first place.

If you want to skip straight to the part where the computer does (mostly) exactly what you tell it to do, might I recommend a programming language.

@pfriedma if you can read faster than you can type, llm might still be useful. Scnr

@pavel
Doubtful. Smaller models (ie trained auto complete ) I'd agree with tho.
The hard part rarely the velocity of typing. IDEs have had syntax completion for a ...long time.
The hard part is understanding the problem sufficiently to describe it accurately in the first place. The parts that LLMs tend to acclerate well are boilerplate generation and the like and, I'd argue the solution there is to eliminate the need for specifying common patterns entirety instead of wasting compute generating it.
Erlang does this pretty well IMO with OTP Behaviors. Other languages have interfaces etc for a similar purpose.

The problem being discussed here is once you've gotten there, explaining it in natural language tends to be too ambiguous.
People are spending more time fighting with getting an LLM to output what they want than it probably would take to just describe what you want in a less ambiguous language designed for the purpose. Like a programming language.

@pfriedma I agree that autocomplete is "obviously good". I would not mind powerful model doing that, so it makes less mistakes.

Getting example code works for me, too. Instead of "lets search the web for something similar", sometimes LLM can produce better starting point.

Plus it is very useful for translating between languages. "for A in `cat txt` do rm -i $A" -- please translate it to python3" was one useful example. Here is 500 lines of Lua / JavaScript, but I need Python for MicroPythonOS was another time saver.

@pfriedma Other example was "Give me Tetris implementation for MicroPythonOS". Sometimes, I don't care about exact implementation, I just want something to start from.

@pavel
I don't disagree that that's a useful capability for a system to have. I don't believe though that LLMs are strictly required to do that. Your examples are, IMO a perfect example of where tailored SLMs would be perfectly suited, be easier to reason about, require fewer resources and less energy - and - be less prone to be relied upon for *everything*.

@pfriedma Aha. No disagreement there; I have easy access to LLMs but no tailored models. Yes, it should be possible to do this stuff with less energy.

(And better integration with emacs would be cool, too. I'm not switching my editors for this, but good autocomplete would be cool.)

@pfriedma I think it would be pretty cool if we could have programming languages which were natural subsets of English (see: HyperTalk, AppleScript, Inform) and which had near-miss helper systems that could outline English sentences the natural-language-programming system can't unambiguously interpret and suggesting similar sentences it could interpret unambiguously.

But you'd still at some point have to teach people to consider a problem logically and break it down into steps.

@mcc @pfriedma I feel like people that want "natural language programming" want it exactly because they want to be less verbose. So IMO a system that make you write out extremely verbose natural language is not really desirable.

@shironeko @pfriedma I think the hard problem in teaching programming is getting people over the hump of "I can't do that". I think syntax, and the need to learn it, is one such hump. The existence of people who can't "program" but can do feats in spreadsheets or flowcharts that sure as heck look like programming seems like evidence for this idea.

I don't *know* if natural language is a third way around it, but I wonder.

There may be other problems also however.

@mcc @pfriedma Yep, the core problem is the crazy number of people who are only to happy to move what little bit of thinking they are currently doing to a machine.

I don't get the allure personally but maybe that's just me.

@shironeko @pfriedma Hence scare quotes. But programming or no, general programming environments and APIs are not set up to accept code in this form. Your sheets in Excel, Google Sheets, or Multimedia Fusion are stuck within those specific proprietary programs and can't readily interact with, say, C libraries the way Python can. The development tool community has never given a fiftieth of the consideration to accommodating spreadsheet programmers as it has to randomly generating code with LLMs

@mcc @pfriedma I count excel as a perfectly normal way to program.

@pavel @pfriedma No matter the cost to people and the environment?

@mcc @pfriedma

> outline English sentences the natural-language-programming system can't unambiguously interpret

i agree this would be super-useful, but: surely in any programming language, _every_ statement only has _one_ way it will be interpreted, and the problem is really that it might not be the way the programmer wants?

i mean, '8 - 6 * 2' might be ambiguous to a human; but not to a compiler…

@fishidwardrobe @pfriedma Presumably this is one of the design considerations you'd have to wrestle with when making such a language

@nurglerider @pfriedma I believe you are seriously overestimating costs of running the model (as opposed to model training).

@pavel @pfriedma And I believe you aren't considering the scale of the majority of humanity using AI and the impact it would have. This is a huge failing of all AI supporters, they love to minimize the impact without thinking about how even tiny impacts add up at scale.

If this weren't a problem we wouldn't have the dangerous proliferation of data centers right now.

@nurglerider @pfriedma Estimate energy needed to produce 500 lines of code by LLM, and then compare to energy human would need for same task.

I could ask LLM to do the computation, but I guess you'd prefer me not to do that :-).

I believe running LLM model is comparable to running game (while the model is running!) and certainly less energy than running a car.

@pfriedma You can’t even satirize this shit anymore because it writes itself

@fishidwardrobe @mcc @pfriedma Unfortunately, many programmming languages do have ambiguous statements; we usually call them "undefined behavior"...

@cesarb @fishidwardrobe @pfriedma We tend to treat those as flaws in the language and phase either the flaws, or the language, out!

@mcc @pfriedma are we counting COBOL as a success or a failure in this regard?

@WizardOfDocs @pfriedma COBOL is obviously one of the most successful programming languages of all time.

I am not qualified to comment off the cuff on whether its level of natural language-ness, compared to the more compact languages that followed, concretely helped its adoption by people who otherwise would have struggled with coding.

The canonical success in this area is SQL, which is both very "natural" and very manager friendly.

@cesarb @mcc @pfriedma i'm going to go out on a limb here: does the compiler or interpreter always behave the same way when encountering a given single example of an "ambiguous" statement?

i bet it does. either the compiler / interpreter rejects the statement; in which case it's a program error. or it interprets the statement in some set way; in which case, that's what the statement does.

either way, it's not actually ambiguous — except to a human.

@fishidwardrobe @mcc @pfriedma Unfortunately, no; when encountering undefined behavior, the way the statement is interpreted is not fixed. It can depend on something very far in the code, the environment, or the phase of the moon. Even compiling the exact same code again with the same compiler can result in different behavior.

@cesarb @mcc @pfriedma well then i'm with @mcc — that's a compiler bug

@fishidwardrobe @cesarb @pfriedma It's not a bug, it's undefined behavior. UB and "compiler-defined behavior" have well-understood meanings in programming, although the behavior itself is by definition not well understandable, nor do C programmers tend to understand how badly they are UBing every time they try to write code.