Inferring TypeScript types #5
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
@glen for our TypeScript hybrid to work, it is essential that we can actually reflect upon the TypeScript interfaces and types and turn them into JavaScript.
I've done a bit of experimentation, and noticed that the support for generics is quite limited, and there is not yet support for type aliases.
There is no support for type alias yet, but there is a (stale) PR which looks very promising: https://github.com/typescript-rtti/typescript-rtti/pull/85
Here is a dump of what I tried so far:
From the experiments I did a while ago, I am not too worried. As I said in one of my long comments a few days ago (that may have gotten lost in the wash), as far as I can tell you can always get the local "syntactic" type of an item, even if it is a type alias (you may not have any way of looking up the definition of the aliases), so I think we can just hardcode in Dispatcher what our aliases mean. That means if we change our aliases we'll also have to patch the code in Dispatcher, another reason for trying to get our definition scheme as solid as possible before diving into the implementation of Dispatcher, which will immediately need rtti...
We agreed that the order of operations here would be to "finalize" the syntax with which we would like to write the implementations, and then as soon as we have the candidate for that, hook up typescript-rtti and dump all the information it provides and see if there's enough for us in JavaScript to compute the dependencies needed, and the parameter types and return type of the implementation.
@glen I've made a little start today with
typescript-rttiin therttibranch (created from theapproach_4.6branch).I'm trying to do some stuff in the file
src/index.ts, so far with almost no success. To be continued...Ah -- I did make a PR of the approach4.6 branch and merge it into main. Could you rebase your branch on main and (force-)push it? Thanks. Then I'll be very interested to take a look!
Ah I missed that, for some reason my local branch
mainwasn't connected to the remotemain. Fixed now.I've renamed the folder with output from
objtobuild, and I've temporarily added apackage.jsonthere whith a"type": "module"field else I can't get the output to run with node (how did you run the output before?).Shrug, not sure why renaming was needed (obj has been standard for 40 years for object files) but on the other hand the name is kind of arbitrary anyway, so no big deal either way.
Since I was reluctant to check in any files in that resided in an output directory, I had just made one in obj by hand, figuring that at some point, either in the prototype or in the development of the NextMathJS (TM), there would be a build script and it would create the package.json as needed. Anyhow, I've now created an issue mentioning this point, thanks should have done that before.
Hmmm, the build-and-run was giving errors for me, and it turns out it's in the first step (the ttsc, output quoted below). Are you seeing the same reports, do you know what they mean, and is it part of the difficulties you mention you are having? Thanks for letting me know.
Ah, that's funny, I've never seen an output folder being named
/obj😅. In the JavaScript ecosystem I mostly see them named/buildor/distso I figured it would make sense to name it like that. I'm glad you're ok with it. It's indeed not a big deal.We should indeed solve the need for
/build/package.jsonin a more neat way, thanks for opening an issue for this to not forget.I get indeed the same errors when I run
npx ttsc -b. Also, I didn't yet manage to get Call site reflection working in our prototype (in the playground of the website of typescript-rtti it works).I guess I am showing my age. In the years when most code was C or C++, obj was completely standard. I do see that now
buildappears to have become more common. But also I read that Visual Studio automatically creates an obj directory, so it's not completely gone yet... anyhow, as we've agreed, it doesn't really much matter so no need to go back and forth.I am worried about those ttsc errors. It may be that rtti is not currently up to the challenges of this code. I will try to look into this next time I have some minutes to on mathjs.
It could be that the combination of the version of rtti and that of typescript is giving problems. I experimented a bit with different versions of TS but that didn't solve it.
Short update:
1f2a59c802CallSiteby configuring TypeScript to output CommonJS instead of ESM, there is a bug (I think) in typescript-rtti that breaks generated ESM code, I've reported that.specificationsobject holding all function factories. It partly works, but I get stuck when trying to get types of thedepsfunction argument of functions. This is essential information for us.To be continued.
Another status update.
I've been experimenting with two approaches:
typescript-rtti, see branchrtti(see the readme in that branch for instructions on how to run). I didn't succeed in getting useful type information out of reflection withtypescript-rtti. Due to limited documentation, it is a lot of trial and error and looking into raw internal data structures thattypescript-rttioutput. I have the feeling thattypescript-rttiwill not solve what we want.experiment/typescript_plugin(see the readme in that branch for instructions on how to run). I didn't get any meaningful type information out of our code yet though. I think it is possible to make this work, there are functions available that should make something like that possible. There is a bit of documentation and some nice examples to get started with the compiler. It will require a lot of work though to work it out.It is very well possible that someone smarter than me can get one of these two approaches working, but I'm basically stuck, this is not something that I'm strong at.
Way forward:
dep: Dependencies<'multiply', T>andSignature<'square', T>, but use plain definitions likedep: { multiply: (a: T, b: T) }. Such that we can extract the type information purely based on the AST or even plain text source code withough needing to interpret and resolve it. This only offers very limited TS support, I'm not sure if that is worth it.OK, I have finally had a chance to start looking at this again. Sorry teaching in last Spring semester and the move back to LA thereafter took up all of my spare time. Anyhow, I just pushed a commit to the
rttibranch that updates all the software and gets it running again (typescript-rtti has switched to thetpsccompiler in place ofttsc, for example) and I agree that the type information being produced by typescript-rtti appears to be incomplete :(I have isolate a fairly simple test case (really the one you came up with) in the file
src/isolate_bug.tswith information how to run it in the README.However, there are other runtime typers out there. The most mature appears to be
@deepkit/type. So I am going to make a new branchdktypeand try that out.If we can't find a runtime typer that works for us, I will file issues against the existing typers and see which one makes us happy first ;-)
Thanks for kicking this investigation off, and once again, sorry for the delay!
Good news --
@deepkit/typeindeed provides accurate runtime type information for the simple bad case isolated from therttibranch. You can see it in action on thedktypebranch (would love verification that it works for you as well). (Note since I branched off ofmainagain, it's back to building inobj; feel free to move that to build again if it will make your life better/easier.)When either of us gets a chance, we can verify whether deepkit will indeed ferret out the type information we actually need from the actual (proof of concept) structures we are using in this approach. If we are fortunate and it does, then I think (presuming you are still interested) we could get this project going again.
Unfortunate news --
@deepkit/typepunts on some of the very complicated generic instantiations we are currently using in typocomath and just characterizes them as typeany. For the "smallest" example I have managed to track down so far, please seesrc/complex/arithmetic.tsin thedktypebranch.As far as other runtime typers go, there is Hookyns/tst-reflect, but that seems currently embroiled in a major redesign, so it does not seem like a good point now to try to adopt it. A recent post from hookyns (https://github.com/Hookyns/tst-reflect/issues/78#issuecomment-1669254012) says it will be "at least a month to get it to alpha," so I'd guess it's at least 2-3 months before there'd be something we can practically try.
There are other typers, e.g. gtx/typescript-rtti (different from typescript-rtti), @tsmirror/reflect, ts-runtime, but their last commits are 4,3,3 years ago, respectively. So I very much doubt any of them are applicable now. (If you want me to try any of them to see if they still function, let me know, but I doubt its a productive use of effort.)
So I am doubtful there at this moment exists a runtime typer that will handle our needs.
Possible ways forward (not all mutually exclusive, some could certainly be pursued in parallel):
File the isolated obstruction from branch
rttias an issue againsttypescript-rttiFurther boil down the obstruction from branch
dktypeto get to something that could practically be filed as an issue against@deepkit/type(this will take a bit of work untangling the typocomath code into a stand-alone example).Instead of/in addition to 1 and/or 2, try to hack on and improve one of these typers to cover our use cases, with the intention to submit PRs to them.
Wait for the next major version of hookyns/tst-reflect and see how that does.
Since typescript introduces a build step to produce javascript anyway, write our own bespoke tool to extract the particular data we need from our .ts code, presumably using the typescript compiler API (in other words, we can leverage the official compiler, there's no need to write our own parser, for example). Then we'd just add a call to that tool to our build scripts. I don't have a clear picture of how much work this would be. I believe your
experiment/typescript_pluginbranch is one possible beginning to this approach.Use one of the vast number of schema definition language packages that let you describe types with (runtime) data structures, which the packages enable typescript to then infer the matching types at compile time from. (These approaches generally involve more or less redundancy between the schema specifications and typescript type declarations, which is one reason people don't love them.)
Roll our own little signature language in the spirit of (6), given that we have a pretty specific subset of schemas we want to specify, basically named collections of (often generic) functions. This is basically m93a's original approach in over.ts, in which he was parsing string signature designators and transforming them into compile-time types. I think it is also the approach you investigated in #7 here. It also has at least some redundancy issues, and maybe some difficulties with generics. So I am not personally enthusiastic about (6) or (7), but one or the other or both is probably viable. I'd say if we were going this route it might be nice to try to re-use (i.e., do (6) rather than (7) if possible).
Back off of implementing in typescript, sticking with JavaScript instead, but do an overhaul of typed-function that automatically generates TypeScript .d.ts files. Note it is true that in this approach, we are definitely creating some little signature language of our own like pocomath or even current typed-function uses; but at least there is not redundancy between those signature strings and a language-internal notation.
If we go route (8), I'd advocate making it a major overhaul/rewrite of typed-function, adding arbitrary cross-implementation dependencies and generics to typed-function at least, as per pocomath, and possibly subtypes as well, for the sake of obtaining the efficiency gains of pocomath and making extensions like adding comprehensive BigInt support significantly easier. (I feel like the aspirational ideal down this road would be to get the coupling loose enough that we could have a collection of packages like @mathjs/number, @mathjs/matrix, @mathjs/bigint, etc, and you get to use the types of the ones that you import... so if you never calculate with units, just don't import @mathjs/units and you won't clutter your symbol space with "ohm" for example)
OK, I know I "just got back" to this project but I think I will pause again until I hear some thoughts from you on which one(s) of these directions the overall mathjs effort ought to try. Well, number (1) is a low-hanging fruit, since we've already isolated what seems to be a behavioral deficiency, so likely I will do that in any case.
OK, (1) from the previous note is done: https://github.com/typescript-rtti/typescript-rtti/issues/102
Actually (2) was not nearly as much work as I worried. I tracked down one specific obstacle in the latest commit to branch
dktypein the pair of filesremote.tsandlocal.ts-- executenode obj/local.jsafter compiling withnpx tsc.I have filed https://github.com/deepkit/deepkit-framework/issues/466
So while those issue submissions are percolating you/we can contemplate which of 3-9 or other options we might pursue if the issues don't come back with a solution. (My guess is that even if they do, we will encounter more obstacles with the respective runtime typer(s) given the complexity of what we are trying to accomplish...)
On (6) in the above list, one of the more extensive schema options is TypeBox (https://github.com/sinclairzx81/typebox). It makes no mention of class types, but has essentially everything else from TypeScript that I could think of, even generics which are critical to us and which very few of the existing schema languages cover (in fact none other that I reviewed). It has a provision for extension of the type system, so it is possible we could extend it to cover classes (since we use them for things like Complex).
The only two other packages I saw that explicitly support generics are ArkType (https://github.com/arktypeio/arktype) and TypeOnly (https://github.com/paroi-tech/typeonly/tree/master) but as far as I can see neither has any support for function types, which is an even bigger necessity for us. The package TypeOnly is interesting in that it uses exactly typescript syntax as its schema language: in other words, you separate out your type-only parts of your code into .d.ts files, you use them normally in your typescript files, and your run the typeonly parser to produce runtime type information. But since it is re-implementing the TypeScript parser, it's (currently) missing many language features (and doesn't seem particularly active). But it is an informative example, sort of a bit between option (5) and (6) above.
So I would say TypeBox is at the moment the leading candidate for (6), if that's a direction of interest.
Today's search (based entirely on https://www.typescriptneedstypes.com/) turned up another runtime typer Typia (https://github.com/samchon/typia) that would work essentially along the same lines as typescript-rtti or
@deepkit/typeand is clearly under active development/maintenance, but it has no support for function types (nor does it seem to have any such support planned).I also took a brief look at your
experiment/typescript_pluginbranch (basically a start on option 5 from my long list) and I totally think that could be made to work. Basically we would identify the expressions where we need to add runtime information and add extra properties or arguments. It's just a matter of whether we want to do the work for this.Imagine for a moment that we didn't even want to insert an extra "infer()" function call as you do. Using the
squareimplementation ingeneric/arithmetic.tsas an example, we would "just" need to recognize this code:(which should fit a fairly specific pattern) and transform it to
Then it will compile, and when the Dispatcher registers the "square" implementation, it just looks up the info it actually needs to know how to instantiate it and inject its dependences in those extra properties we added to the function object.
So heck, we could use most any standard macro language running prior to the typescript compilation step to do this interpolation for us (at least if we added a macro call around the export). The killing downside of that otherwise easy-to-implement plan is we'd almost certainly lose the IDE support on our source files because they would likely stray from properly-typed valid TypeScript. So that seems out.
So we want to take advantage of the fact that the code is all correctly typed TypeScript without the interpolated "Runtime data" -- it just won't actually behave correctly without that data filled in. And this "Runtime data" will all be vanilla JavaScript. So we would just want a step either before, as part of, or after the typescript compilation that extracts the information we need and inserts the extra lines. If we think that extraction will need the full TypeScript parse, then of course a plugin is the way to go -- we're definitely not writing another parser for TypeScript!
But if we think the pattern can be kept really standardized, though, it might actually be easier to write a little purpose-built tool that scans the code and inserts the extra lines at whatever point in the build process is easiest. (We can of course introduce an extra function call that acts as a no-op to flag the places we want it to happen; we would just need to properly type that function call and implement it as the identity in our core, so that everything in the IDE will still just work.) I suppose to avoid additional dependencies that tool should be written in TypeScript or JavaScript, even though my first impulse would be to use Python. Adding such a tool does feel hacky, but I am not sure it's truly any more of a hack than even something like typescript-rtti...
Anyhow, that's the summary of my thoughts on option (5).
OK, I have a proposal on how to proceed if you'd like to stay with TypeScript implementation of the next major revision of mathjs without the need for a runtime typer that works on our cases (i.e. to avoid waiting for one or needing to enhance an existing one ourselves).
It is a hybrid of our current "approach 4.6" and your proposed "approach 6" (per #7). So far I have only made a working version of this for one instance, the generic "square" implementation that we often use as an example. You can see it running in branch
supply_runtimeand also providing access to the dependencies and signature at runtime.In this instance, it ends up replacing approach 4.6's
with
So: we would have to choose concrete, runtime-available JavaScript values to encode the dependencies and their types and the signature. We were always going to need such encodings, we were just assuming they would come from the runtime typer, and we'd just adapt to whatever that package returned. Instead, we will have to choose that encoding ourselves, and so we will have to try to keep it as simple and obvious and straightforward as possible, to minimize the extent to which we are replicating the TypeScript type definition language (but we will surely need bits of it, unfortunately).
(Every runtime typing library actually does this, too -- the key advantage with the runtime typer would be that the implementation typings would simply be written in TypeScript, and the JavaScript runtime representations would be generated from it and handled internally in the core code, so they would never be exposed to someone just adding a new function to mathjs. To avoid the runtime typer, someone implementing a new operation will have to learn our encoding for specifying the dependencies and their types etc.)
Anyhow, once that choice is made (and above I basically just replicated the existing arguments to
Dependencies<...>andSignature<...>for the encoding), then the genericImplementation function (which is itself actually a generic function, but it deduces the type parameters from its arguments) must be supplied with the literal entities that encode the necessary information. It then statically infers the type of the dependency object that must be supplied and the type of the function that needs to be returned, and so deduces the type of its final argument that gives the actual function body of the implementation. Because that type ends up being deduced in detail, the body needs no redundant type information specified, but it is still being fully type-checked as the examples insrc/index.tsin this branch show. Then genericImplementation annotates that function body with the dependency and signature encodings it has been supplied with, for use at runtime, as also shown in index.ts.So, as you point out in #7, there ends up being just one source of truth, the literal objects specifying the dependencies and signature. This version does work with generics. That's what the "generic" in "genericImplementation" refers to; I envision also a "specificImplementation" function that works just for one concrete type, e.g.
specificImplementation('number', {complex: 'number'}, sqrt, [IMPLEMENTATION BODY])for the number implementation of sqrt that also depends on the operationcomplex.And for specific implementations with no dependencies I would suggest something like
groundImplementation('number', 'add', (a, b) => a+b)where again we can avoid reiterating the type information on the function body because ininterfaces/arithmetic.tswe specified thataddwas(a: T, b: T) => T(which we will also have to modify to describe the types via some literal encoding for runtime, and deduce the actual TypeScript type ofaddfrom it).Although not as streamlined as approach 4.6 by itself would be if we had a working runtime typer, because it necessitates implementors directly using our encoding of dependencies and signatures, this plan does avoid both redundancy and any extra build step. If we strive to keep the encoding as clean and simple and close to TypeScript's own notation, I think it should end up reasonable to use.
There are of course many other details to work out but I will wait on all that until our discussion.
Just reading up on all we wrote last time...
Some very short first thoughts:
supply_runtimebut didn't see the types actually inferred yet, could be someting stupid on my side though.Oh, and one more thought:
(5.) What we would like to do is use macros in our code base. For example https://www.sweetjs.org/ offers this. As far as I can see, Sweet.js only works with JS and not with TS. I do not see how we can integrate something like this to our case, but maybe it sparks some ideas on your side :).
Oh maybe this is something: https://github.com/blainehansen/macro-ts
Responses to your latest 1. - 5.:
well one big clue will be whether/how fast either runtime typer responds to the issues I filed.
one stupid-ish thing we could do easily is write a buildstep that would insert the line after the "// capture" comment below:
and then we'd have to write a parser just strong enough to handle all the implementation declarations we actually saw in practice. That doesn't sound terribly hard...
I mean are there existing tools for runtime access to source code comments?
did you check out
supply_runtime, dopnpm_install, thennpx tsc, thennode obj? it prints to stout that the dependencies of square are{multiply: 'T'}so voila, that's the info we need.Yes at first glance it looks very much like macro-ts would solve our situation. let me know if you think (a) you would be willing to go for the goofy syntaxes and (b) you don't think it will ruin the ide experience you are seeking. if so would be happy to whip up a PoC with it.
I am not seeing any existing facility for grabbing comments at runtime.
but during my search I did see that built in to current typescript that there are class decorators that optionally will emit type information about their invocations. (unfortunately no plain function decorators, dunno why :-P) should we think about/try artificially wrapping each of our implementations in a class with a decorators to try to get type information out that way?
Looking a bit more at macro-ts i see no commits for almost two years and
let me know if these are too big of red flags for you. All the necessary functionality is advertised there. But who knows if it works with Typescript 5.1?
Yeah, not a good idea to start using an inactive project.
I did a bit of experimenting in the
experiment/typescript_pluginbranch to read JSDoc comments. The TypeScript compiler reads those also into an AST. So we can use that though it is just as easy to read "real" TS types from the AST, so no real benefit.I found some really clear documentation and examples of TypeScript transforms:
https://github.com/itsdouges/typescript-transformer-handbook#transforms
https://github.com/itsdouges/typescript-transformer-handbook/tree/master/example-transformers
So indeed, it looks like it is not that hard to write our own TypeScript plugin that replaces some empty placeholder (like
// capture) with the inferred types. I would like to work out this experiment further.I'm not sure how the class decorators would work out, but it can be interesting.
About
supply_runtime: ah, yes, I can run it now, though I have to create a temporarypackage.jsoncontaining{"type":"module"}to be able to runnode obj, don't you have that issue?What I mostly mean though is that I can compile with
npx tsc, but it looks like we don't actually get compile time safety: I can change the code to the following and it still compiles, wheras it should give an error:oh dear. i guess the type inference I set up lost information -- those two examples fail in main, is that right? and yes, I forgot to put the "make a package.json in onj step" this time, sorry.
on macros i also found https://github.com/GoogleFeud/ts-macros
that is up to the minute and seems to have the features we need. it doesn't typecheck the generated code but that is OK because we just want to assign literals to properties, typechecking would be trivial anyway. no discussion either way about effects on ide. probably easier than writing our own transformer -- it has a built in $$typetostring macro already, for example. if you don't get to it first, I will give it a try since it looks like my supply_runtime example didn't work =(
Hurray! I think we have hit pay dirt. Thank you for your perseverance in looking for good solutions, and for bringing up the idea of macros. I have just pushed branch
ts-macroswhere I have successfully used thets-macrospackage I linked in the last note to reflect the type of the generic square implementation, while leaving its body fully type-checked. Note there's one extra installation step you need to do, see the README.I think this is the way to go. It's basically identical to if typescript-rtti had worked the way we wanted it to out of the box.
In this very first PoC, the syntax where the implementation is defined is
and the reflected type ends up being the string literal
{ <T>(dep: Dependencies<"multiply", T>): (a: T) => T; reflectedType: string; }But ts-macros definitely seems flexible enough to do either/both of the following (A) and (B) as desired:
(A) Streamline the syntax to something like
or
or possibly even something decorator-like:
(B) change the structure that's passed back from a string to some other encoding we devise. Not sure if this one is worth trouble or if we should just parse the string. It has all the info we need, although I think it is worth checking if the info is there at macro expansion time to further resolve
Dependencies<"multiply", T>and pass us back that information as well. It might be convenient; on the other hand, we can just reflect the global Signatures interface and look it up in there as well ourselves at runtime.Cheers! Looking forward to hearing your thoughts. Seems plausible we could settle on this when we meet. In the meantime, let me know if you want to try (A) and/or (B) or want me to. Thanks!
On one part of (B) just mentioned, indeed I found on the TypeScript issues tracker a trick that will look up Dependencies for us at compile time. It's easy enough that it is probably worth using. I've checked it in to branch
ts-macros. As it's a bit of a trick, we can either get the argument type transmogrified into being the return type, i.e.or we can get the implementation rendered in its actual structure, just with "deep expansion cursors" left at all of the generic parameter leaves of the type expression:
of course, if we reflect these as string literals, which is probably the path of least resistance, it will be trivial to rewrite all of the
DeepExpand<T>s back to justTto get:which has just the information we want.
So it seems like this is really coming together. I will see if I can continue the
ts-macros branch to wrap reflection up in the middle syntax of (A) above as that seems the most streamlined and also if we need to do any other bookkeeping on a per-implementation basis (even just counting how many implementations are included in a given Dispatcher, say), it will give us one place to do it. (If so we might rename it from
$reflect!to something more encompassing like$imp!or$handle!or... But I will leave it as$reflect!for now since that's all we'd be using ts-macros for to begin with. Have a good weekend!Oh, I forgot, the "deep expansion" trick differs depending on whether the implementation is generic or concrete. And according to the TypeScript guru jcalz on StackExchange, there's no way to test a type to tell which it is! So I will provide two macros, which for now I will call
$genericRT!and$concreteRT!(because we need to wrap every implementation in one or the other, and$genericReflect!and$concreteReflect!are getting so darn long...OK I think I have fully investigated what's possible with
ts-macros; all of the options are running in the latest commit to the corresponding branch. I will recap my findings here.The experiments are all with the generic implementation for
square; recall in main we have the strong typing working but no type reflection, and the relevant code is:With
ts-macrosyou can add the following line to get detailed type info:but that's clearly too much boilerplate to include with every implementation. So we need to wrap it up in a macro. So I made one where we could write
and get some type info that way, without an additional line. But it turns I couldn't get the long crazy trick to do the detailed type info lookup to work inside a macro (due to TypeScript funkiness with generic function types), so you get out just
which really is enough information, but would require some additional lookup at runtime in another table (that we can hopefully reflect from the definition of the
Signaturesinterface). Or, by inserting one more generic type wrapper we can do the lookup at compile time:yields
<T>(dep: { multiply: (a: T, b: T) => T; }) => (a: T) => Twhich is exactly what we want.Finally, the author of ts-macros is working on features that should soon allow us to streamline this even a bit further to
if we want. Not sure if this is clearer/simpler than the explicit
export const square... version; saves about a dozen characters per import, which is pretty minimal.On the other syntax options I mentioned above, the label
$reflect:before the declaration turns out not to work, and I didn't pursue the$reflect!(square)one after the declaration because I didn't see any reason to repeat the identifier.So anyhow, when we meet we can discuss whether to use ts-macros, and if so, which syntactic option to use, where to store the type info (currently putting it on a
.reflectedTypeproperty, but maybe we should use thereflect-metadatamodule that is designed for this kind of purpose, and how to proceed from here.That sounds promising 😎 , let's discuss the options.
A thought:
ts-macrosis a TypeScript plugin. If we look up what$typeToStringis doing under the hood, it may be just as easy to write our own TypeScript plugin (so we have full control and no dependency).Personally I don't see enough benefit to that to outweigh the intricacy and lack of good documentation of the Typescript Compiler API. If we decide to go the plugin route, we could create a ts-macros plugin that provides the info we need, and then if you at any point decide to write a leaner plugin that does just only exactly what we want, we should be able to switch over and jettison ts-macros without much difficulty. I am definitely more interested in getting on with the trajectory of mathjs than in wrestling with the guts of TypeScript. So if someone else has gotten a plugin working that will do what we need, I am happy to reuse. The fact that neither of two fairly mature reflection plugins actually covered our use cases only makes me more concerned about going the route of rolling our own.
Very good point, me too 😄
Just got a response (https://github.com/deepkit/deepkit-framework/issues/466#issuecomment-1697584538) that @deepkit/types is not able or at least not going to support type-only imports, which are necessary for the typocomath scheme to work. So that possible vehicle for RTTI appears to be off the table permanently.
Ok clear, thanks for the update.
Here a short recap of our meeting last week:
ts-macrosjust works like a charm to extract the types and make them available runtime.ts-macrosuses to extract type information. If that is straightforward, create our own TypeScript plugin.typocomathPOC with full functionality similar topocomathcontaining a set of functions and a couple of datatypes, working for real.Right, I put all that in #10 (as it seemed larger in scope than this issue).
Ahh, sorry. I didn't receive a notification about #10, turns out I was not watching
typocomath(now I do). I still have to read your comments.Ugh, we are not out of the woods with ts-macros yet. I got it working very nicely with generic
square, but then I went to the other end of the spectrum with Complexsqrtand hit https://github.com/GoogleFeud/ts-macros/issues/74 and https://github.com/GoogleFeud/ts-macros/issues/73.Let's see how the ts-macros guy responds. He's been very responsive. But my bug report wasn't the best. If we can't get him to fix these things up, we may have no choice but to try to do our own plugin or parsing literal strings into types.
🤔 that is a bummer. Let's see indeed if this can be fixed in
ts-macros.Is there a way to work around it? If this is a special edge case, and 90% of the functions work just fine, it may be acceptable if it cannot be handled the easy way. We could fallback to using a JS type definition instead if these are edge cases. Far from ideal, but just trying to be pragmatic.
FYI: I'm planning to work on the POC coming Thursday.
As far as working around the current limitations of ts-macros, if you look at mathjs as it stands, complex sqrt is not particularly complicated. There are numerous functions with many more dependencies and much longer function bodies. So I would say if complex sqrt is not working, there's no chance of that type reflection mechanism being feasible in general. I'll let you know if I hear anything from the ts-macros maintainer.
If not, what will you work on on Thursday? Either the plugin you created or the literal string type parsing? Of the two, I like the idea of your plugin better. I wouldn't worry too much if it doesn't "decode"
Dependencies<'multiply', T>intomultiply(a:T, b:T) => T: for one, we have all of the information needed to do that decoding ourselves at runtime, and for another, in my local ts-macros branch, I have rephrased theDependenciesgeneric type so that TypeScript seems more apt to always "decode" it itself. So it really seems like your plugin could be a viable way to go. You could see if it has the issue with ellipses in long type names, and if so, if you can find a way to prevent the ellipsization from taking place. Just some thoughts.I guess you're right.
It makes sense to do some more fiddling with the custom TypeScript Plugin, I'll do that (unless there is other news by then). Indeed the basic information is there. I will start by updating the
Dependenciesgeneric type to see whether that makes a difference already.OK, I put the latest version of the Dependencies template into branch ts-macros-issues (I didn't check it into ts-macros because ts-macros still isn't working; I don't think there's anything wrong with the Dependencies template).
Good news, GoogleFeud has purportedly fixed both the issues I filed against ts-macros in his repo. On a project of this timescale (months/years), my plan was just to wait until his next release to test (he seems to release pretty often). So if you do work on this tomorrow, you can contemplate whether you prefer to work on your plugin, or perhaps figure out how to work from his repo rather than an npm release and just test and see if it fixes branch ts-macros-issues as it stands, in which case maybe it's not worth your while putting more time into your plugin, up to you.
Ok today I've done the following:
mkdirandcpworking Windows even with pnpm shell-emulator, could be that I'm doing something wrong there)ts-macrosfrom git directly instead of npm so we can use the bug fixes (the fixes are not yet published on npm)reflectedTypestring into a structured object, so we can use that object in the dispatcher to actually do something. It is very bare-bone for now but seems to work quite nicely, I'll post some outputs in the next comment.I've been working in the
ts-macros-issuesbranch.Here the console output for three different functions. It looks like we have all the information that we need!
I was thinking about how it would be easiest to create a typocomath similar to pocomath (and working for real). I was thinking about these steps:
ts-macros-issuesbranch intomain/srcinto/src-temp/srcinto typocomath/src, make that compile and run (so that is still just JS, but working for real)/src-tempinto the right place in/src.reflectedTypeproperty, and make it work for real$implementmacroDoes that make sense @glen ? If so, I think I can do large part of this work next week on Thursday, or would you like to give this a shot before Thursday?
Great! That's a nice chunk of work. If you're comfortable just moving ahead with ts-macros for type reflection and putting the other two alternatives aside for the time being, then that's cool with me.
Great thanks. Don't understand why this wasn't working -- I thought one of the points of pnpm was it provided a cross-platform scripting environment -- but this is fine too, the point is just to get things working.
Yes good for testing but should we wait until there's a properly-published npm package before merging into main?
Nice! Output looks promising. Needs some minor touch-up like clearly indicating whether the operation is generic and if so what the template parameter is (I think we are limiting to one parameter for now). Or maybe we are just assuming the parameter is always
T?Hmm, looks more roundabout than I was imagining. Plus I think by far the biggest piece is writing the guts of the dispatcher. I think that should be from scratch with the new set of behaviors in mind (i.e., I don't think typed-function as it exists will be too helpful). So I was just going to dive into that and try to get everything in this POC right now actually working first, before trying to expand to everything in pocomath. I'd like to wait until ts-macros is released with the fixes before merging and diving into that. I will keep you closely updated, and of course if I don't get to it til Thursday you are welcome to embark how you see best. But if type reflection is settled, probably best to continue this discussion in #10.
Yeah sorry I should have posted in #10, I'll continue there...