this is my kid in a candy store wishlist of programming language features that are kicking around in my head:
- small and consistent
- functional w/eg lexical scope and closures
- mutablity allowed
- managed memory
- statically typed with full inference
- record types w/row polymorphism
- polymorphic variants
- multiple dispatch
- coroutines
- multiple backends, including its own VM
- well-defined interface to the host
- single threaded
admittedly some of these these are in tension 🤔
small and consistent is critical because I want a language that fits in my head. this is the main draw of lisp, and my original conception of this language was a lisp but I'm not 100% committed to that anymore.
functional is non-negotiable to me. lexical scope, first class functions, anonymous local data structures, closures etc are the best building blocks to software I've ever used.
I say this in contrast to conventional object oriented programming which is, frankly, some non-sense with no theoretical underpinning. there are a few projects out there similar to what I'm thinking that are held back by their commitment to these debunked antiquated ideas.
that said I tend to want to make video games and things for which a very particular kind of performance is critical, and forcing immutable data structures and their overhead is a non-starter unfortunately. I know this from experience... in my experience I avoid the kinds of bugs you get from mutable state by keeping mutation local with coroutines and control flow, and by keeping this whole thing single threaded.
managed memory is critical because that's just not the level of abstraction I want to be working at when I am implementing creative ideas. it feels like trying to paint with a Ziploc bag full of loose bristles instead of a paint brush.
that said there is not really a one size fits all solution to memory. in my thinking this is a hosted language and questions about memory are moved to the embedding interface. for the VM it would be a combination of some reference counting the papers I've read.
I have not become more conservative politically as I've gotten older, but I have become more sympathetic to static typing. I do not tend to run into state bugs in practice but I *do* run into bugs that would have been prevented by a better type checker...
that said unlike a lot of type nerds I do not think annotations and declarations are "inherently good" and I'm much more drawn to systems that let me elide them and/or allow anonymous types. Roc really sent me off down this path.
related to that is my more recent reading on records with row polymorphism, which is what I think want out of a record type. I want to be able to create them anonymously wherever I need them and pass them around while not abandoning checking. JavaScript objects come *so close* but they have a bunch of weird edge cases around properties that are unfortunate, and TypeScript is limited in its checking and inference around them...
polymorphic variants are the row polymorphic version of algebraic data types which I found transformative to use in rust and f#. they offer such a nice compromise where an expression can be "one of a set of values" which to me feels like a really powerful midpoint between "every expression is an instance of one type" and "anything can be anything". exhaustive pattern matching is also really nice.
the classic type theory stuff really falls down around dispatch though. every function, generally, can only have one signature for type theoretic reasons. that means you end up with functions with names like map2 which is gross. more critically operator overloading becomes impossible, and in video games I'm doing linear algebra... all the time. so many examples of type inferencers take addition to be defined as an operation on numbers where in my world I need it to support vectors matrices etc
it's not just classical typed languages, this is also my biggest gripe with making video games in JavaScript. if I have to write Vector3.addition instead of + the languages barely usable for linear algebra. this kind of feature gets brushed under the rug generally because I guess most programmers are not doing that much arithmetic? the Julia language is my guiding light here
coroutines! absolutely non-negotiable feature for me to feel expressive making interactive software. I wandered the deserts of PLT looking for semantics I could paint with and I found them.
they require a compiler transformation to do right so the language needs to support them at a fundamental level. I expect multiple dispatch + static typing + coroutines is going to run into formal roadblocks but I'm willing to dream.
from my experience in clojure I'm wary about languages with multiple backends but I think it's worth pursuing. particularly if this language is explicitly designed to be hosted and as a result makes few assumptions about its host. once type checking and other transformations are done code generation to different backends is not intractable. this one is maybe a nice to have idk
I have thoughts for a standalone VM that would support this language too, probably something like a rust library. my thinking uses reference counting for managed memory and I think you can set it up to be more or less pauseless, basically by deferring decrement and release operations into chunks of work that can be run "in between" events while the system is idle. I also have a sketch of spaghetti stacks to implement the coroutines.
given that it is a hosted language then the interface to the host is critical. this is what makes Lua and Python attractive as embeddable languages.
the model I have in mind is more similar to the language Céu where the VM exposes very constrained inputs and outputs, almost like a shader but for behavior/state machines. it's basically an event loop that allows coroutines to block on events that the host can unblock, optionally passing in event data.
finally: it is absolutely single threaded, which often gets dismissed as a deficiency but the benefits you gain from simplicity and, critically, determinism cannot be overstated. it is so important to be able to build reproducible code when developing video games and interactive software. the ability to capture and replay input and reproduce state reliably is critical. having everything run on one thread makes that possible.
@nasser ah ok that makes more sense!!
@nasser @ivan the way love2d does this where you can have a separate isolated instance of the VM that technically runs in the same process but it's a different thread and you just communicate over queues has been really good when I've used it
you have concurrency still, but it's "outside" the language; you can't have shared access to state across threads but that feels like a feature
run all the simulations you want in the background and they just communicate actor-style
@technomancy @ivan honestly im happy to write the networking/physics or whatever in rust/c#/javascript/hostlang, those arent the "interesting" bits of the system. thats just software engineering and traditional tools are fine to me, not the subjective artistry of making a state machine that drives an experience for a human being.
but js does this kind of concurrency and i think its good. the one downside is data serialization costs but that can be avoided if the system is designed right.
@nasser @ivan (it was really funny discovering this coming straight from an erlang job and an FP background because it was like ... "wait a minute ... you're game devs .... how did you figure out that this was the right solution? you're supposed to be imperative programmers using an imperative language")
@technomancy @ivan more and more game dev stuff is encountering fp because of its utility! the last gdc i attended before covid had a talk on the then new spiderman playstation game and how they used immutable data structures in their asset pipeline to make asset versioning/revisions possible ✨
@ivan oh 100%, to me that all lives in the host and is passed into the state machine that this language implements like uniforms to a shader. the way it affects and updates game state is what i want to be single threaded and deterministic, the actual mechanism of computing, sending, receiving that data is not the target of this system. thats the thinking anyway, it likely has holes and limitations.