Change Dispatcher into a builder function that returns a plain object (of lots of function properties) #19

New Issue

Open

opened 2023-10-17 22:27:28 +00:00 by glen · 16 comments

glen commented 2023-10-17 22:27:28 +00:00

Owner

Copy Link

Right now the way you actually use a Dispatcher is left a little fuzzy. Does it put operations on itself, i.e:

const math = new Dispatcher(allSpecifications)
assert(math.add(2, 4) === 6)

Or does it have some method like bundle that returns the object with all the operations?

const dispatcher = new Dispatcher(allSpecifications)
const math = dispatcher.bundle()
assert(math.add(2, 4) === 6)

Or something else?
This needs to be pinned down, and then the type of the resulting object of operations needs to be set up (based on the literal type of allSpecifications).

Right now the way you actually use a Dispatcher is left a little fuzzy. Does it put operations on itself, i.e: ``` const math = new Dispatcher(allSpecifications) assert(math.add(2, 4) === 6) ``` Or does it have some method like `bundle` that returns the object with all the operations? ``` const dispatcher = new Dispatcher(allSpecifications) const math = dispatcher.bundle() assert(math.add(2, 4) === 6) ``` Or something else? This needs to be pinned down, and then the type of the resulting object of operations needs to be set up (based on the literal type of `allSpecifications`).

glen added the

question

label 2023-10-17 22:27:56 +00:00

josdejong commented 2023-10-18 12:44:51 +00:00

Collaborator

Copy Link

If we implement the Dispatcher as a class, we should not attach functions to itself, then we mix things up.

However, the end user is probably not interested in a verbose const math = new Dispatcher(allSpecifications).bundle(), so I think we will end up creating a helper function so you can use it like const math = create(allSpecifications).

I'm OK with writing the Dispatcher as a class if you have a preference for that, though for these kind of cases I normally use closures (like in the makeCounter example here) rather than a class, so I have plain and simple state and functions rather than a code base full of this and methods attached to this.

If we implement the Dispatcher as a class, we should not attach functions to itself, then we mix things up. However, the end user is probably not interested in a verbose `const math = new Dispatcher(allSpecifications).bundle()`, so I think we will end up creating a helper function so you can use it like `const math = create(allSpecifications)`. I'm OK with writing the Dispatcher as a class if you have a preference for that, though for these kind of cases I normally use closures (like in the [makeCounter example here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures#emulating_private_methods_with_closures)) rather than a class, so I have plain and simple state and functions rather than a code base full of `this` and methods attached to `this`.

glen commented 2023-10-18 17:14:12 +00:00

Author

Owner

Copy Link

OK. I looked back at pocomath, and there indeed I did attach functions to instances of the PocomathInstance class as implementations were added, so exactly what "we should not" do. Oops!

I am fine with refactoring this to have just a plain function createDispatcher that takes an allSpecifications (and eventually an additional optional argument that is a dispatcher) and returns a "dispatcher" which is just a plain object with lots of function properties, one for each operation. This "dispatcher" object will need some additional properties like its configuration, but also it will need to remember its specifications so they can be folded into a future dispatcher built from it, and I am sure there will be some other things like that. For now I will just prefix the names of any such things that aren't in the "user namespace" with an underscore to avoid name clashes, and we can just ban operations that start with underscore.

Changing the title of this issue to reflect this design.

OK. I looked back at pocomath, and there indeed I did attach functions to instances of the PocomathInstance class as implementations were added, so exactly what "we should not" do. Oops! I am fine with refactoring this to have just a plain function `createDispatcher` that takes an allSpecifications (and eventually an additional optional argument that is a dispatcher) and returns a "dispatcher" which is just a plain object with lots of function properties, one for each operation. This "dispatcher" object will need some additional properties like its configuration, but also it will need to remember its specifications so they can be folded into a future dispatcher built from it, and I am sure there will be some other things like that. For now I will just prefix the names of any such things that aren't in the "user namespace" with an underscore to avoid name clashes, and we can just ban operations that start with underscore. Changing the title of this issue to reflect this design.

glen changed title from design: How will the operations of a Dispatcher be called? to Change Dispatcher into a builder function that returns a plain object (of lots of function properties) 2023-10-18 17:15:41 +00:00

glen added

refactor

and removed

question

labels 2023-10-18 17:15:58 +00:00

glen referenced this issue 2023-10-18 17:30:50 +00:00

Trajectory for Typocomath #10

glen commented 2023-10-18 17:35:15 +00:00

Author

Owner

Copy Link

Note that one early major step in this refactor will be specifying the proper TypeScript return type of createDispatcher -- it has to combine all of the types of all of the implementations of each operation into the type of the corresponding property of the return value. So it will be a major piece of TypeScript type language coding. I think the original TypeScript p.o.c. of a typed-function-like-thing that m93a did will be quite instructive here.

Note that one early major step in this refactor will be specifying the proper TypeScript return type of createDispatcher -- it has to combine all of the types of all of the implementations of each operation into the type of the corresponding property of the return value. So it will be a major piece of TypeScript type language coding. I think the original TypeScript p.o.c. of a typed-function-like-thing that m93a did will be quite instructive here.

glen commented 2023-10-19 06:41:41 +00:00

Author

Owner

Copy Link

OK I have pushed branch dispatcher_refactor with a start on this. Changing from a class to a builder that returns a closure was no problem of course. However, I have run completely aground on the task of specifying the return type of the createDispatcher function. What I pushed gets the property names of the return value exactly correct, as you can see by looking at the generated index.d.ts -- it filters out the types that are being installed, and ignores the aliases. And it does combine the types of all of the implementations. The problem is that given an implementation like
square = <T>(dep: {multiply: (a: T) => T}) => (a: T) => T, we want the type of the resulting square property on the created dispatcher object to be <T>(a:T) => T. Right now, it's producing

    square: {
        <T_2>(dep: {
            multiply: (a: T_2, b: T_2) => T_2;
        }): (a: T_2) => T_2;
        reflectedType: string;
    };

i.e the dependency is still in place. You might well think that all we want is ReturnType<> of the type that it's currently producing for square. But sadly, ReturnType<<T_2>(dep: {multiply: (a: T_2, b: T_2) => T_2;}): (a: T_2) => T_2;> is (a: unknown) => unknown, not <T>(a: T) => T, as I think it should be. I have been unable to think of any way to pass from the factory types to the types of the functions they produce after about a half-day of beating my head against it, so I have asked https://stackoverflow.com/questions/77321309/version-of-returntype-that-preserves-genericity. This appears to actually be a major roadblock to a fully TypeScript-native version of typocomath...

On the other hand, one can see that the desired type transformation is a pretty simple textual transformation: just delete everything after the template parameter (if any) up to and including the first =>. That's all. So we could still pursue the avenue of using something like the typing I just pushed, basically writing the innards of Dispatcher in JavaScript, and then adding a build step that does the text transform on the generated .d.ts file to produce the .d.ts to go along with the shipped .js module produced. Annoying and hackish, but would likely work, and save us most of the work of manually writing type definitions.

Or maybe there is a way to go from types like <T>(dep: {multiply: (a: T) => T}) => (a: T) => T to <T>(a:T) => T that you or someone on StackOverflow will come up with. But once again I don't see how to proceed until we find/select some solution to this: ultimately the whole point of this exercise was to come up with an automatic way to generate exactly the type of the generated "mathjs object" (here called the "dispatcher") with all of the proper property (i.e. operation) names and their types. The index.d.ts I have generated is tantalizingly close to this, but the last kilometer to go here in that journey is currently staunchly eluding me...

OK I have pushed branch dispatcher_refactor with a start on this. Changing from a class to a builder that returns a closure was no problem of course. However, I have run completely aground on the task of specifying the return type of the `createDispatcher` function. What I pushed gets the property names of the return value exactly correct, as you can see by looking at the generated index.d.ts -- it filters out the types that are being installed, and ignores the aliases. And it does _combine_ the types of all of the implementations. The problem is that given an implementation like `square = <T>(dep: {multiply: (a: T) => T}) => (a: T) => T`, we want the type of the resulting `square` property on the created dispatcher object to be `<T>(a:T) => T`. Right now, it's producing ``` square: { <T_2>(dep: { multiply: (a: T_2, b: T_2) => T_2; }): (a: T_2) => T_2; reflectedType: string; }; ``` i.e the dependency is still in place. You might well think that all we want is `ReturnType<>` of the type that it's currently producing for `square`. But sadly, `ReturnType<<T_2>(dep: {multiply: (a: T_2, b: T_2) => T_2;}): (a: T_2) => T_2;>` is `(a: unknown) => unknown`, not `<T>(a: T) => T`, as I think it should be. I have been unable to think of any way to pass from the factory types to the types of the functions they produce after about a half-day of beating my head against it, so I have asked https://stackoverflow.com/questions/77321309/version-of-returntype-that-preserves-genericity. This appears to actually be a major roadblock to a fully TypeScript-native version of typocomath... On the other hand, one can see that the desired type transformation is a pretty simple _textual_ transformation: just delete everything after the template parameter (if any) up to and including the first `=>`. That's all. So we could still pursue the avenue of using something like the typing I just pushed, basically writing the innards of Dispatcher in JavaScript, and then adding a build step that does the text transform on the generated `.d.ts` file to produce the .d.ts to go along with the shipped .js module produced. Annoying and hackish, but would likely work, and save us most of the work of manually writing type definitions. Or maybe there is a way to go from types like `<T>(dep: {multiply: (a: T) => T}) => (a: T) => T` to `<T>(a:T) => T` that you or someone on StackOverflow will come up with. But once again I don't see how to proceed until we find/select some solution to this: ultimately the whole point of this exercise was to come up with an automatic way to generate exactly the type of the generated "mathjs object" (here called the "dispatcher") with all of the proper property (i.e. operation) names and their types. The index.d.ts I have generated is tantalizingly close to this, but the last kilometer to go here in that journey is currently staunchly eluding me...

glen commented 2023-10-19 16:20:20 +00:00

Author

Owner

Copy Link

Hmm, well, the responses to the StackOverflow question above were fairly discouraging. I mean, the guru jcalz showed a way of doing it for a single generic function, but he confirms the analogous thing for every property of a plain object fails.

The links from that question of its "duplicates" show that classes sort of work to do a similar thing, but I am not really sure how to cook up a class with the methods we want and then how to extract the type of what we want createDispatcher() to return.

And then in fact I tried interfaces and again, we can get close to what we want, without the class baggage, but again there's just a single generic parameter to the whole interface, not a generic in each property. But in this case, it is possible to move the generic from the interface to each property value! You can see this at

https://www.typescriptlang.org/play?jsx=0&noErrorTruncation=true&ts=5.2.2#code/FAEwpgxgNghgTmABAMwK4DsIBcCWB7dRABzjxFQjAGcAeYRRAOUTAA8sx0QrEqs4c6AOYAaRPUQBBFu07dEAbwDaAaUSCmAXQBciGOgCeSzQF8xEgEIyOXHsrUbGOvYeNnxDAErW5d1esInXX0DEwA+AAoJBiFdewCtXQiAOlSYXUlVTQBKRABeMMQU1IAjXQss3ILETyyTbLj-R2di5LLECpUc-MLarpNgQfBoeCQIAj5EAAsARjjolDw8XRoAFUjwIjiAW1QoLCT0xFWxMtWqwtX6nqL085uTjwZEEvgki9vtdFRtkrA4D58ATCAbjdCTADWM3yxFI5EoVAis2ywAA9KjEFA8FRqIgsAYiEhBMg8HBtjBcAQANyIKHJEl4aYwHj4wlFVJtYKubrVEIMbQAWiioEgsAQiBGVB4ABEftsDGtCgoJAykpsdnsDp9jqdtPdqlcPhE7h9Vk9XnB3jdjV8fn8ATcgYIhMABqykIwALKKmEQsAGPDIRCy7byxXAd2IADq+iwPryiiagW962cnjAWFQcHQqwJYBoIbD6yyYQG6MQAAlmXipkhI8TSeTKehTqgsIgAO5IdBgMAgPGMsEAN3+7ZwY-QWEZMGIsA0eBKACtIO2O+OpjWwGiMUJOP8cBAeFQpjhkBx+4IpywYBANyQ8IS4PixOhB1BUESg1hawgAOQ8GcOxgAwB0QEBGW-CkXjbdQsG3Xhe22HgMCgHA-SgAwOUGS9-mQG8kCmH1lQYVUinVRRdn2Q5dEeM5TWuaobX1S5zAYC0rUY9Jvl+f5AX4Z1XUGSMAEkvXjWl-UDaZwxEmNJ3E+JHBTMI0wzLMczzGhCOLLpS3g1YTx4HAWVrAIODgPDKE7JAgMnMQ2EoIh2xgIQYA0LsmRHPReGdKAkF3HsBAgRACE3TspjwPz4Jwiz8JfPBnMIGLLJs9dvIC-dgtCsAbzvUhH3xLDI3TGAoAwuT2wTRTkxoDAIVfDt0BUlZ1laeAhCoXQAAV4BgbYM3+WhhIqxUSw+dNM2zXNCRoYbY1G3STCAA

... maybe this might be the beginning of a possible solution: the type of ReallyWant is pretty much exactly what we really want. Will try to look at this more later.

Hmm, well, the responses to the StackOverflow question above were fairly discouraging. I mean, the guru jcalz showed a way of doing it for a single generic function, but he confirms the analogous thing for every property of a plain object fails. The links from that question of its "duplicates" show that classes sort of work to do a similar thing, but I am not really sure how to cook up a class with the methods we want and then how to extract the type of what we want createDispatcher() to return. And then in fact I tried interfaces and again, we can get close to what we want, without the class baggage, but again there's just a single generic parameter to the whole interface, not a generic in each property. But in this case, it _is_ possible to move the generic from the interface to each property value! You can see this at https://www.typescriptlang.org/play?jsx=0&noErrorTruncation=true&ts=5.2.2#code/FAEwpgxgNghgTmABAMwK4DsIBcCWB7dRABzjxFQjAGcAeYRRAOUTAA8sx0QrEqs4c6AOYAaRPUQBBFu07dEAbwDaAaUSCmAXQBciGOgCeSzQF8xEgEIyOXHsrUbGOvYeNnxDAErW5d1esInXX0DEwA+AAoJBiFdewCtXQiAOlSYXUlVTQBKRABeMMQU1IAjXQss3ILETyyTbLj-R2di5LLECpUc-MLarpNgQfBoeCQIAj5EAAsARjjolDw8XRoAFUjwIjiAW1QoLCT0xFWxMtWqwtX6nqL085uTjwZEEvgki9vtdFRtkrA4D58ATCAbjdCTADWM3yxFI5EoVAis2ywAA9KjEFA8FRqIgsAYiEhBMg8HBtjBcAQANyIKHJEl4aYwHj4wlFVJtYKubrVEIMbQAWiioEgsAQiBGVB4ABEftsDGtCgoJAykpsdnsDp9jqdtPdqlcPhE7h9Vk9XnB3jdjV8fn8ATcgYIhMABqykIwALKKmEQsAGPDIRCy7byxXAd2IADq+iwPryiiagW962cnjAWFQcHQqwJYBoIbD6yyYQG6MQAAlmXipkhI8TSeTKehTqgsIgAO5IdBgMAgPGMsEAN3+7ZwY-QWEZMGIsA0eBKACtIO2O+OpjWwGiMUJOP8cBAeFQpjhkBx+4IpywYBANyQ8IS4PixOhB1BUESg1hawgAOQ8GcOxgAwB0QEBGW-CkXjbdQsG3Xhe22HgMCgHA-SgAwOUGS9-mQG8kCmH1lQYVUinVRRdn2Q5dEeM5TWuaobX1S5zAYC0rUY9Jvl+f5AX4Z1XUGSMAEkvXjWl-UDaZwxEmNJ3E+JHBTMI0wzLMczzGhCOLLpS3g1YTx4HAWVrAIODgPDKE7JAgMnMQ2EoIh2xgIQYA0LsmRHPReGdKAkF3HsBAgRACE3TspjwPz4Jwiz8JfPBnMIGLLJs9dvIC-dgtCsAbzvUhH3xLDI3TGAoAwuT2wTRTkxoDAIVfDt0BUlZ1laeAhCoXQAAV4BgbYM3+WhhIqxUSw+dNM2zXNCRoYbY1G3STCAA ... maybe this might be the beginning of a possible solution: the type of ReallyWant is pretty much exactly what we really want. Will try to look at this more later.

glen commented 2023-10-19 20:08:58 +00:00

Author

Owner

Copy Link

Have thought about this a bunch more. Let me summarize where we are at. Right now in trying to type the dispatcher, we have something basically like

type Implementations = {
  foo: <T>(dep: {dependency: (a:T, b:T) => T}) => (a:T) => T)
  bar: () => (a: number) => string
}

and we want to get to the type

type Operations = {
  foo: <T>(a: T) => T
  bar: (a: number) => string
}

The playground above shows how to get from the type

type GenericImplementations<T> = {
  foo: (dep: {dependency: (a:T, b:T) => T}) => (a:T) => T)
  bar: () => (a: number) => string
}

to Operations. So GenericImplementations seems like a natural stepping stone. But I just can't seem to find a way to generate GenericImplementations<T> from Implementation after beating my head against it for a while. So I have asked https://stackoverflow.com/questions/77326673/is-there-a-way-to-lift-the-level-of-genericity-of-an-object-whose-values-are-g. If that doesn't come back with any leads and we can't figure it out, then I see four ways to proceed:

1. Figure out some different way to get from type Implementations to type Operations that doesn't go through GenericImplementations -- as I said, the responses to the question I asked on that were not promising, but there's a lot of material in the questions that were linked to it as "duplicates" (they're similar, but I don't think really duplicates, but not worth fighting that.) Nevertheless, I have low confidence of success on this route, but it still seems conceivable despite the discouraging response.

2. Gather up the type GenericImplementations from all of the implementation files as we go along. This would involve adding basically redundant expressions at the bottom of each file. For example in Complex/arithmetic.ts, we would add something like

export type GenericImplementations<T> = {
  add: typeof add<T>, addReal: typeof addReal<T>, unaryMinus: typeof unaryMinus<T>, conj: typeof conj<T>,
  subtract: typeof subtract<T>, multiply: typeof multiply<T>, absquare: typeof absquare<T>, 
  divideReal: typeof divideReal<T>, reciprocal: typeof reciprocal<T>, divide: typeof divide<T>, sqrt: typeof sqrt<T>
}

Since this is very boilerplate-ish, possibly we could get the reflect macro or another similar macro to do it along with the reflection it's already doing.

3. Basically switch to JavaScript at the point where we've deduced the type Implementations, having Typescript write out the .d.ts at that point, and then do the relatively straightforward textual manipulation on that file that will produce the definition of the Operations type, and just use that modified file as the .d.ts file we ship with the bundled module.

4. Stop all this painful wrestling with TypeScript, basically go with pocomath but keep things constrained to operations that we can capture with TypeScript types and write code to generate a .d.ts file from the pocomath operation descriptions, so we don't need to generate one by hand. (So again, no "numint" type as there is in pocomath because TypeScript can't actually support that type, even if all we are doing is providing .d.ts files describing a plain JavaScript implementation.)

As you can see, I am getting a little fatigued with the long struggle to bend TypeScript to our will, but am happy to continue with any one (or I suppose even more than one) of the above options, or anything else you may suggest. I'll pause on things here until I hear some feedback.

Have thought about this a bunch more. Let me summarize where we are at. Right now in trying to type the dispatcher, we have something basically like ``` type Implementations = { foo: <T>(dep: {dependency: (a:T, b:T) => T}) => (a:T) => T) bar: () => (a: number) => string } ``` and we want to get to the type ``` type Operations = { foo: <T>(a: T) => T bar: (a: number) => string } ``` The playground above shows how to get from the type ``` type GenericImplementations<T> = { foo: (dep: {dependency: (a:T, b:T) => T}) => (a:T) => T) bar: () => (a: number) => string } ``` to Operations. So GenericImplementations seems like a natural stepping stone. But I just can't seem to find a way to generate `GenericImplementations<T>` from `Implementation` after beating my head against it for a while. So I have asked https://stackoverflow.com/questions/77326673/is-there-a-way-to-lift-the-level-of-genericity-of-an-object-whose-values-are-g. If that doesn't come back with any leads and we can't figure it out, then I see four ways to proceed: 1) Figure out some different way to get from type Implementations to type Operations that doesn't go through GenericImplementations<T> -- as I said, the responses to the question I asked on that were not promising, but there's a lot of material in the questions that were linked to it as "duplicates" (they're similar, but I don't think really duplicates, but not worth fighting that.) Nevertheless, I have low confidence of success on this route, but it still seems conceivable despite the discouraging response. 2) Gather up the type GenericImplementations<T> from all of the implementation files as we go along. This would involve adding basically redundant expressions at the bottom of each file. For example in Complex/arithmetic.ts, we would add something like ``` export type GenericImplementations<T> = { add: typeof add<T>, addReal: typeof addReal<T>, unaryMinus: typeof unaryMinus<T>, conj: typeof conj<T>, subtract: typeof subtract<T>, multiply: typeof multiply<T>, absquare: typeof absquare<T>, divideReal: typeof divideReal<T>, reciprocal: typeof reciprocal<T>, divide: typeof divide<T>, sqrt: typeof sqrt<T> } ``` Since this is very boilerplate-ish, possibly we could get the reflect macro or another similar macro to do it along with the reflection it's already doing. 3) Basically switch to JavaScript at the point where we've deduced the type Implementations, having Typescript write out the .d.ts at that point, and then do the relatively straightforward textual manipulation on that file that will produce the definition of the Operations type, and just use that modified file as the .d.ts file we ship with the bundled module. 4) Stop all this painful wrestling with TypeScript, basically go with pocomath but keep things constrained to operations that we can capture with TypeScript types and write code to generate a .d.ts file from the pocomath operation descriptions, so we don't need to generate one by hand. (So again, no "numint" type as there is in pocomath because TypeScript can't actually support that type, even if all we are doing is providing .d.ts files describing a plain JavaScript implementation.) As you can see, I am getting a little fatigued with the long struggle to bend TypeScript to our will, but am happy to continue with any one (or I suppose even more than one) of the above options, or anything else you may suggest. I'll pause on things here until I hear some feedback.

glen commented 2023-10-19 21:10:40 +00:00

Author

Owner

Copy Link

Yeah, as I worried, TypeScript guru jcalz is pretty sure the transformation from Implementations to GenericImplementation is impossible... so it seems like it's one of (1) through (4) or something else you may come up with. Looking forward to your thoughts.

Yeah, as I worried, TypeScript guru jcalz is pretty sure the transformation from Implementations to GenericImplementation<T> is impossible... so it seems like it's one of (1) through (4) or something else you may come up with. Looking forward to your thoughts.

glen commented 2023-10-20 00:32:38 +00:00

Author

Owner

Copy Link

Hmm, I guess there may be a variant of option 2 that might lead to less redundancy:

2b) Specify all of the implementations in each file as members of a class with one generic parameter. That way the types and definitions can be specified together. So far as I can tell, classes are the only entities that can be generic and contain both types and the values of those types. So this will avoid specifying all of the functions, and then putting all of their types into a generic interface (or equivalently we could specify all of the types, and then type the implementations using the generic interface). The catches here are that there is some baggage with classes that we won't be using, and the classes won't merge automatically the way that contents of modules do when you import them and the way that interfaces do when you reopen them. So we will have to put some small amount of code into each of those aggregator modules that currently just import and re-export a bunch of stuff, in order to merge the interfaces of the classes.

But this 2b option can work in principle, I'll post a long playground link at the bottom of this comment that shows it running to combine Implementations<T> and Other<T>, which we imagine are in different files, via Joined<T> which we imagine is in one of those aggregator modules, to eventually get down to exactly the GenericImplementations<T> template we want (although it's called MoreImpls<T> in this playground), which we know we can convert to the exact dispatcher type we need as described above.

I'm not necessarily saying that 2b is the way we want to go; it is simply another option. It's a bit ugly to be stuffing things in classes, but it doesn't add any more redundancy and at least we know it can definitely work to type the eventual dispatcher object. But the original option (2) might be better as it avoids the baggage of classes and I am pretty sure we can get it to work. It's just verbose, but maybe we can reduce that verboseness with ts-macros -- not really inclined to try unless we decide to go that route.

Example of (2b) in action: https://www.typescriptlang.org/play?jsx=0#code/MYGwhgzhAECSC2AHEBTeKB2AXMWCWA9hhADwAqAfNAN4CwAUNNAGYEEBc0AFACYqKdqfRJj4ZgAT05cw7MgBpoAI05kAlNAC8VMgF8N27rOjqtOrd2EGqM69GEA6YaMySZ8sGoZMlYAE7SdjKcGACu8EoofnYQWH54GADmFlxBnmbQAOQQBJnQANTQYAy6DAygkDAA8lgAFlHkVHSMyv6BGcHQsfFJdmERUSlpdmAOqEl13q0AXtLCgkTAKNKypoZ6aXJ2ZClWHemGjosovPwOx7Ze9KX0DAlYUcxgS9AAUgQJKDyNNFN4SJwEMg0JgcPgiKRKFMCHVODV6n5GiUGFgJCJoABhMAgcBKVApByE-yJCCcMAYCQjClleio9EAIUgKCqIj8uEIxBIVSUACsqJopjQANoAaWgCWgAGsUBICMxoNyeQBddiChW80VK6AoAAeDwwPBgWJxYDxKDVTGgAH5oAAlFBYUJ+DBkNEoLkakVKihCzLYvCQKrMTJa3X6w3cboJZIAH2goQNKGYnx4V0t6ctNowKAAblELRnOGKw6IYFGktboGKQrn8y1C9Bs3m-ILdELpbL5YqlTS6ShoAARPAQRC4YAIllRdkQkgAcX5guoovFGClMrl0FnKugjIgzNZ085s81FDbHY3W97bugADkwOhIfzB8PR1hx1FJ2zwZz3inGhQr3RAA1bFQhQWBsCiPdgG-ekJBFGUAFUMA5chFBMBCJG1PVSzXTsTCfZomCFAAFFcMJlbcuElVRSKVNRtBzD4eBKaAhTITDQxwg0YC4ajOASZhBlgBiKCYvBU0rWBoBrZtAP7Xd9ynb8AGU8ESDBcCdd1BUVeRBTvdBsPDGBFM-Q9SEVAD6CfECQDAiCHj8aDYPgpCUKIEhF2XCVzy7XkVTVRVNWM3DjVxVAC0rYtuIjQzzXrDNK3tR1nVdEQPR5E8oqYTgUqdF03Uyk9fX9QNgy4kzuHi6A4wTPhk2zVMcozG18rSorgq9KhOATSUMAIAB3DAot6jB+qGkb610fT6z89U+XkhUD1U9TNNSnT61nWbLRqkseOfEcxwnFaOVIecGFs0DwMg5yUBgjk4Mw5DUO8sVfPXeUt1VRK9ti0ymXM79zuyxLMx3QHTqINSNK0vx3WPL1FHi6ykpk+NxoG4aZsFeaLtuWlryHI63wRRzHmeBGF3rJcWXI4nX3fPwgbOucKG3FnobWuGEcUFlTyWgBZAh4aBEBHwsBnjqicm-CeJYSF-Jr-yWgd+AlzQmB2faIz6rHVxtag2Pe1d5rIbcyBC3R0abKIAG4lrFjXB3VkgxZBbALJVgm+2gAAJMA8wsIiWDYThGlOAQaHsfgXHEKQjFURQVBMbZoH0fZVG2KZfACaAI82VOMnLRJkXoPgKnh6BgAhLBoFqQPln9xuHfoGviDrwbySwQWwGlPwLDQihUg6I3WA4evG4ccfGmTtpJ7zBxc5+fRW99gB1bufk19rCoyvsNy77Be-7-9W4AenPy0AD0rSAA

Hmm, I guess there may be a variant of option 2 that might lead to less redundancy: 2b) Specify all of the implementations in each file as members of a class with one generic parameter. That way the types and definitions can be specified together. So far as I can tell, classes are the only entities that can be generic and contain both types and the values of those types. So this will avoid specifying all of the functions, and then putting all of their types into a generic interface (or equivalently we could specify all of the types, and then type the implementations using the generic interface). The catches here are that there is some baggage with classes that we won't be using, and the classes won't merge automatically the way that contents of modules do when you import them and the way that interfaces do when you reopen them. So we will have to put some small amount of code into each of those aggregator modules that currently just import and re-export a bunch of stuff, in order to merge the interfaces of the classes. But this 2b option can work in principle, I'll post a long playground link at the bottom of this comment that shows it running to combine `Implementations<T>` and `Other<T>`, which we imagine are in different files, via `Joined<T>` which we imagine is in one of those aggregator modules, to eventually get down to exactly the `GenericImplementations<T>` template we want (although it's called `MoreImpls<T>` in this playground), which we know we can convert to the exact dispatcher type we need as described above. I'm not necessarily saying that 2b is the way we want to go; it is simply another option. It's a bit ugly to be stuffing things in classes, but it doesn't add any more redundancy and at least we know it can definitely work to type the eventual dispatcher object. But the original option (2) might be better as it avoids the baggage of classes and I am pretty sure we can get it to work. It's just verbose, but maybe we can reduce that verboseness with ts-macros -- not really inclined to try unless we decide to go that route. Example of (2b) in action: https://www.typescriptlang.org/play?jsx=0#code/MYGwhgzhAECSC2AHEBTeKB2AXMWCWA9hhADwAqAfNAN4CwAUNNAGYEEBc0AFACYqKdqfRJj4ZgAT05cw7MgBpoAI05kAlNAC8VMgF8N27rOjqtOrd2EGqM69GEA6YaMySZ8sGoZMlYAE7SdjKcGACu8EoofnYQWH54GADmFlxBnmbQAOQQBJnQANTQYAy6DAygkDAA8lgAFlHkVHSMyv6BGcHQsfFJdmERUSlpdmAOqEl13q0AXtLCgkTAKNKypoZ6aXJ2ZClWHemGjosovPwOx7Ze9KX0DAlYUcxgS9AAUgQJKDyNNFN4SJwEMg0JgcPgiKRKFMCHVODV6n5GiUGFgJCJoABhMAgcBKVApByE-yJCCcMAYCQjClleio9EAIUgKCqIj8uEIxBIVSUACsqJopjQANoAaWgCWgAGsUBICMxoNyeQBddiChW80VK6AoAAeDwwPBgWJxYDxKDVTGgAH5oAAlFBYUJ+DBkNEoLkakVKihCzLYvCQKrMTJa3X6w3cboJZIAH2goQNKGYnx4V0t6ctNowKAAblELRnOGKw6IYFGktboGKQrn8y1C9Bs3m-ILdELpbL5YqlTS6ShoAARPAQRC4YAIllRdkQkgAcX5guoovFGClMrl0FnKugjIgzNZ085s81FDbHY3W97bugADkwOhIfzB8PR1hx1FJ2zwZz3inGhQr3RAA1bFQhQWBsCiPdgG-ekJBFGUAFUMA5chFBMBCJG1PVSzXTsTCfZomCFAAFFcMJlbcuElVRSKVNRtBzD4eBKaAhTITDQxwg0YC4ajOASZhBlgBiKCYvBU0rWBoBrZtAP7Xd9ynb8AGU8ESDBcCdd1BUVeRBTvdBsPDGBFM-Q9SEVAD6CfECQDAiCHj8aDYPgpCUKIEhF2XCVzy7XkVTVRVNWM3DjVxVAC0rYtuIjQzzXrDNK3tR1nVdEQPR5E8oqYTgUqdF03Uyk9fX9QNgy4kzuHi6A4wTPhk2zVMcozG18rSorgq9KhOATSUMAIAB3DAot6jB+qGkb610fT6z89U+XkhUD1U9TNNSnT61nWbLRqkseOfEcxwnFaOVIecGFs0DwMg5yUBgjk4Mw5DUO8sVfPXeUt1VRK9ti0ymXM79zuyxLMx3QHTqINSNK0vx3WPL1FHi6ykpk+NxoG4aZsFeaLtuWlryHI63wRRzHmeBGF3rJcWXI4nX3fPwgbOucKG3FnobWuGEcUFlTyWgBZAh4aBEBHwsBnjqicm-CeJYSF-Jr-yWgd+AlzQmB2faIz6rHVxtag2Pe1d5rIbcyBC3R0abKIAG4lrFjXB3VkgxZBbALJVgm+2gAAJMA8wsIiWDYThGlOAQaHsfgXHEKQjFURQVBMbZoH0fZVG2KZfACaAI82VOMnLRJkXoPgKnh6BgAhLBoFqQPln9xuHfoGviDrwbySwQWwGlPwLDQihUg6I3WA4evG4ccfGmTtpJ7zBxc5+fRW99gB1bufk19rCoyvsNy77Be-7-9W4AenPy0AD0rSAA

glen commented 2023-10-20 03:22:09 +00:00

Author

Owner

Copy Link

Wow, actually that last step in (2b), of going from the generic interface to the concrete interface where each property has generic function type (I'll call that "lowering genericity"), turned out to be far more difficult in a more real case than in the first demo. The hitch was that the Parameters and ReturnType built-in generics collapse an intersection of function types to just one of the functions in the intersection. (That's documented, and I had forgotten about it.) Therefore, it's necessary to do the lowering genericity before coalescing the individual implementations of an operation into the overall type of that operation.

But I could only get lowering genericity to work on a flat generic interface, where each property is directly a function type, rather than on a nested interface where property types are objects mapping names to function types.

So the only order of operations I could get to work was:

(A) Take the return type of all of the leaves of original nested generic interface of implementations (gets rid of the dependency part of the implementation function types)

(B) Flatten that generic interface to a generic interface of function types by coalescing the two levels of keys, separated by underscore (i.e. {numbers: {add: F1, conj: F2}, Complex: {add: F3, conj: F4}} becomes {numbers_add: F1, numbers_conj: F2, Complex_add: F3, Complex_conj: F4}

(C) Lower the genericity, so we have a flat concrete object of generic function types

(D) Parse all those TYPE_OPERATION flattened keys to restore the two-level nested structure we originally had, but now as a concrete type all of whose leaves are generic functions

(E) Coalesce the types of the operations with the same name as before, to get the actual desired dispatcher type, which has the type of each property as typically an intersection of generic function types, one for each implementation of that operation.

Kind of crazy, but it actually works. Note that I don't think there will be any significant difference in this between approaches (2) and (2b). I have (2b) completely working now, but I think (2) will (only) work basically the same way, with steps analogous to (A) through (E).

Fully working playground of (2b), including "calling" the resulting dispatcher object: https://www.typescriptlang.org/play?jsx=0#code/MYGwhgzhAECSC2AHEBTeKB2AXMWCWA9hhADwAqAfNAN4CwAUNNAGYEEBc0AFACYqKdqfRJj4ZgAT05cw7MgBpoAI05kAlNAC8VMgF8N27rOjqtOrd2EGqM69GEA6YaMySZ8sGoZMlYAE7SdjKcGACu8EoofnYQWH54GADmFlxBnmbQAOQQBJnQANTQYAy6DAygkDAA8lgAFlHkVHSMyv6BGcHQsfFJdmERUSlpdmAOqEl13q0AXtLCgkTAKNKypoZ6aXJ2ZClWHemGjosovPwOx7Ze9KX0DFgSItAAIvyklBYm0CgAHliiMKEMABrDAEADuGGgAH4aNAANoAaWgCWgQJQEgIzBMAF1VIjsdBdNAQigAG5RADcZXoCT+fmYYCW0AAUgQEigeI0aFM8EhOC9EKQEMg0JgcPgiG8KBQpgQ6vzXiQavU-I0Zdc7g8UNAAMJgEDgJSoFIOU3+RIQThgDASEY26n3R4AIUgKCqIj8uEIxCVSgAVlRNFMaIjkZC0RisVV-bjg9Bo378V9fv9dfrDag40xodAAEooLChPwYMha32JhHYihwzL6vCQKrMTIEn5-DA8GBcboJZIAH2ggL4zHZPCu2fH2ZhGDJUSzE84SNbqe7SRzSJJ5L8c+J0Gnm+DujhEcx8ZjDq1zzwEEQuGAKvdUS9kpIAHFA8HqKGUcesS-cdAXQgN0PSfH0X3xChDx-aA-3PR4ADkwHQKULCeK8bywO8ogfT0JR9VkRzVODtQAWTARARB4fNC2LAA1fVQhQV9A2gD84R1MNUXRE8-0EL9w24392Oxf9wJ1bEkyXdsYD1A0wCNbUYWoosSzLMSJMrKgNyiXQbk1R5YGwBp3k0aAyIojllLohimII6dOUodVHW1NDYi5UzPw479BLgIzVUof9PyRbzI18ulGmEooYAAAwAEmoHVdAAfXipEADIujiHtdGi-9DPCyhhPxXT9O1AAxcAsD+ByWIFUh6JARj8qiIDgDwp0JARdEAFUMG9EhXKwRpFGg5r-OlJyLwq3BEOQixoOmqqORIQEQXBDBJseRaX0wCxPM4xbZpQCB-0aLhTQcc1LWgAAFfwkILFqSEW6qIvEig7Cs1SRGeyrXsK96bmc6AAGVQmYYdvnIRQjuTNsO0ynpEhYnYpIRuLqCO5L4oSZhBhBnKcxBnc91nehgbBpRFpIbqYYeuHlyypIWP2kKT26qLQfByGSB1On0AoWMWl1BnpOgDGsZS6hccGBFCZhbrIu0vwgYvF5+B2yEPPYzjRr8lbgVBCFBYVQUSEp6ntswRQdWlPTyYvBqmr81r2s6nq+qIaGmDILqJFFhHoJM7kWjhG7ON99F-y4IE8Ru7E1G0Uk2R4Ep4UjiQWxTMWuBjzgZb8OBE4oZO8FHHNYBJmct1uB3nVdHDQJBvBEgwXAiyY4ME3kYNYbRmBAOAx88NIBN1RYp2UDG13vQ6v3ev6tjgoE0KEyF8cE0k7OEdkjMUG3GFF23mAju3JgYU3ysz53S+JNrEB6wgRtmwDztYf7QcUGHeyxwnP+c1vjuVaRsMDbk4MA9aB4e7C2gmPYi8YQJ4Wbq3dufhO7CxfNA7Mfdj6XmvLee8iDvSkDfAwCeNlp4oDarPd2EgF5eyXrrHyvE4w4PhgPBuRDnzgU0gfACnDh7emQW3GiTEeHYn5igdU-9wGG3WroLBTBoKkNrsDVyGEsJ+DGgyJYzEtBsXdJxdRBDsJcLAibBBgiiDCNQWIxQ7pILwJIgQNBwoQAoVMsYzCKptGMlsmyeyRE67aidMdLAbiYCePQiYrRfkdFMXVogTW6pyiSiwF0Ag6AIl7SJJAACYSImpOIOktBEBQggHSaZHIWSkDuIcL4PwXAABEtQ8BNKuAwAA9J07MABafp-ToDAFCOklU2oBm9PgRE9yzxFRuNFNgUCUoUnBOgAACTAOSPaUxWAcGgGdeYsJnDtlcFIIwqhFAqBMNsQkmxrkZDIFMBpnAzp3LWFQFciQSgMD4BUNBQy0nQFqJs5Y6yQVUnoMAQFYJrRYDImiQuploYUFSB0agLA2CcGBeSBwuzhqtACECkF9T-Bcn0BC4GAB1WFMyvqlh+s5E8MLsDwuMtKCF3TswAD0oRAA

Wow, actually that last step in (2b), of going from the generic interface to the concrete interface where each property has generic function type (I'll call that "lowering genericity"), turned out to be far more difficult in a more real case than in the first demo. The hitch was that the Parameters and ReturnType built-in generics collapse an intersection of function types to just one of the functions in the intersection. (That's documented, and I had forgotten about it.) Therefore, it's necessary to do the lowering genericity _before_ coalescing the individual implementations of an operation into the overall type of that operation. _But_ I could only get lowering genericity to work on a flat generic interface, where each property is directly a function type, rather than on a nested interface where property types are objects mapping names to function types. So the only order of operations I could get to work was: (A) Take the return type of all of the leaves of original nested generic interface of implementations (gets rid of the dependency part of the implementation function types) (B) Flatten that generic interface to a generic interface of function types by coalescing the two levels of keys, separated by underscore (i.e. `{numbers: {add: F1, conj: F2}, Complex: {add: F3, conj: F4}}` becomes `{numbers_add: F1, numbers_conj: F2, Complex_add: F3, Complex_conj: F4}` (C) Lower the genericity, so we have a flat concrete object of generic function types (D) Parse all those `TYPE_OPERATION` flattened keys to restore the two-level nested structure we originally had, but now as a concrete type all of whose leaves are generic functions (E) Coalesce the types of the operations with the same name as before, to get the actual desired dispatcher type, which has the type of each property as typically an intersection of generic function types, one for each implementation of that operation. Kind of crazy, but it actually works. Note that I don't think there will be any significant difference in this between approaches (2) and (2b). I have (2b) completely working now, but I think (2) will (only) work basically the same way, with steps analogous to (A) through (E). Fully working playground of (2b), including "calling" the resulting dispatcher object: https://www.typescriptlang.org/play?jsx=0#code/MYGwhgzhAECSC2AHEBTeKB2AXMWCWA9hhADwAqAfNAN4CwAUNNAGYEEBc0AFACYqKdqfRJj4ZgAT05cw7MgBpoAI05kAlNAC8VMgF8N27rOjqtOrd2EGqM69GEA6YaMySZ8sGoZMlYAE7SdjKcGACu8EoofnYQWH54GADmFlxBnmbQAOQQBJnQANTQYAy6DAygkDAA8lgAFlHkVHSMyv6BGcHQsfFJdmERUSlpdmAOqEl13q0AXtLCgkTAKNKypoZ6aXJ2ZClWHemGjosovPwOx7Ze9KX0DFgSItAAIvyklBYm0CgAHliiMKEMABrDAEADuGGgAH4aNAANoAaWgCWgQJQEgIzBMAF1VIjsdBdNAQigAG5RADcZXoCT+fmYYCW0AAUgQEigeI0aFM8EhOC9EKQEMg0JgcPgiG8KBQpgQ6vzXiQavU-I0Zdc7g8UNAAMJgEDgJSoFIOU3+RIQThgDASEY26n3R4AIUgKCqIj8uEIxCVSgAVlRNFMaIjkZC0RisVV-bjg9Bo378V9fv9dfrDag40xodAAEooLChPwYMha32JhHYihwzL6vCQKrMTIEn5-DA8GBcboJZIAH2ggL4zHZPCu2fH2ZhGDJUSzE84SNbqe7SRzSJJ5L8c+J0Gnm+DujhEcx8ZjDq1zzwEEQuGAKvdUS9kpIAHFA8HqKGUcesS-cdAXQgN0PSfH0X3xChDx-aA-3PR4ADkwHQKULCeK8bywO8ogfT0JR9VkRzVODtQAWTARARB4fNC2LAA1fVQhQV9A2gD84R1MNUXRE8-0EL9w24392Oxf9wJ1bEkyXdsYD1A0wCNbUYWoosSzLMSJMrKgNyiXQbk1R5YGwBp3k0aAyIojllLohimII6dOUodVHW1NDYi5UzPw479BLgIzVUof9PyRbzI18ulGmEooYAAAwAEmoHVdAAfXipEADIujiHtdGi-9DPCyhhPxXT9O1AAxcAsD+ByWIFUh6JARj8qiIDgDwp0JARdEAFUMG9EhXKwRpFGg5r-OlJyLwq3BEOQixoOmqqORIQEQXBDBJseRaX0wCxPM4xbZpQCB-0aLhTQcc1LWgAAFfwkILFqSEW6qIvEig7Cs1SRGeyrXsK96bmc6AAGVQmYYdvnIRQjuTNsO0ynpEhYnYpIRuLqCO5L4oSZhBhBnKcxBnc91nehgbBpRFpIbqYYeuHlyypIWP2kKT26qLQfByGSB1On0AoWMWl1BnpOgDGsZS6hccGBFCZhbrIu0vwgYvF5+B2yEPPYzjRr8lbgVBCFBYVQUSEp6ntswRQdWlPTyYvBqmr81r2s6nq+qIaGmDILqJFFhHoJM7kWjhG7ON99F-y4IE8Ru7E1G0Uk2R4Ep4UjiQWxTMWuBjzgZb8OBE4oZO8FHHNYBJmct1uB3nVdHDQJBvBEgwXAiyY4ME3kYNYbRmBAOAx88NIBN1RYp2UDG13vQ6v3ev6tjgoE0KEyF8cE0k7OEdkjMUG3GFF23mAju3JgYU3ysz53S+JNrEB6wgRtmwDztYf7QcUGHeyxwnP+c1vjuVaRsMDbk4MA9aB4e7C2gmPYi8YQJ4Wbq3dufhO7CxfNA7Mfdj6XmvLee8iDvSkDfAwCeNlp4oDarPd2EgF5eyXrrHyvE4w4PhgPBuRDnzgU0gfACnDh7emQW3GiTEeHYn5igdU-9wGG3WroLBTBoKkNrsDVyGEsJ+DGgyJYzEtBsXdJxdRBDsJcLAibBBgiiDCNQWIxQ7pILwJIgQNBwoQAoVMsYzCKptGMlsmyeyRE67aidMdLAbiYCePQiYrRfkdFMXVogTW6pyiSiwF0Ag6AIl7SJJAACYSImpOIOktBEBQggHSaZHIWSkDuIcL4PwXAABEtQ8BNKuAwAA9J07MABafp-ToDAFCOklU2oBm9PgRE9yzxFRuNFNgUCUoUnBOgAACTAOSPaUxWAcGgGdeYsJnDtlcFIIwqhFAqBMNsQkmxrkZDIFMBpnAzp3LWFQFciQSgMD4BUNBQy0nQFqJs5Y6yQVUnoMAQFYJrRYDImiQuploYUFSB0agLA2CcGBeSBwuzhqtACECkF9T-Bcn0BC4GAB1WFMyvqlh+s5E8MLsDwuMtKCF3TswAD0oRAA

glen commented 2023-10-20 03:50:44 +00:00

Author

Owner

Copy Link

Oh, in looking into whether I could generate the needed interface for option (2) via a macro (ideally folded into the same $reflect! macro we're already using), I discovered that ts-macros has a $$typeMetadata! built-in macro that returns type information in a structured way, at least for some types. It might make a better version of $reflect! that would prevent needing to parse the results of $$typeToString! as we're currently doing. We should probably try switching at some point and see how well it works in our case.

Oh, in looking into whether I could generate the needed interface for option (2) via a macro (ideally folded into the same `$reflect!` macro we're already using), I discovered that ts-macros has a `$$typeMetadata!` built-in macro that returns type information in a structured way, at least for some types. It might make a better version of `$reflect!` that would prevent needing to parse the results of `$$typeToString!` as we're currently doing. We should probably try switching at some point and see how well it works in our case.

glen referenced this issue 2023-10-20 13:55:41 +00:00

Trajectory for Typocomath #10

josdejong commented 2023-10-20 14:25:02 +00:00

Collaborator

Copy Link

Hmm, all these new TS issues suck, we definitely don't want that.

The $$typeMetadata! sounds promising!

In my head, Typocomath is an extension on top of Pocomath. I think the step to generate type definitions during a build step is something we need in Pocomath too, and it can be based on using JS signatures. In the case of Typocomath, we "only" add a new, second way to define and feed functions to the dispatcher: TS functions, which during a build+reflection step are turned into JS functions with a .reflectedType attached to them. These can be processed by the Dispatcher in exactly the same way as "pocomath" JS functions. If we want we can let the reflection step generate objects matching the current pocomath API (instead of a function with .reflectedType attached), so the Dispatcher doesn't even know whether a function orginally was TS with an auto generated signature, or a manually typed JS function.

Maybe this schematic picture helps explain what I mean:

If I understand you correctly, you're trying to automatically let the TS compiler output the TypeScript definitions (yellow block in the picture, generating index.d.ts), but I think we should generate ourselves during a build step, using a "generateTSDefinitions" function that we implement ourselves in the Dispatcher (yellow block in the picture).

Shall we schedule a video call to think this through?

Hmm, all these new TS issues suck, we definitely don't want that. The `$$typeMetadata!` sounds promising! In my head, Typocomath is an extension on top of Pocomath. I think the step to generate type definitions during a build step is something we need in Pocomath too, and it can be based on using JS signatures. In the case of Typocomath, we "only" add a new, second way to define and feed functions to the dispatcher: TS functions, which during a build+reflection step are turned into JS functions with a `.reflectedType` attached to them. These can be processed by the Dispatcher in exactly the same way as "pocomath" JS functions. If we want we can let the reflection step generate objects matching the current pocomath API (instead of a function with `.reflectedType` attached), so the Dispatcher doesn't even know whether a function orginally was TS with an auto generated signature, or a manually typed JS function. Maybe this schematic picture helps explain what I mean:  If I understand you correctly, you're trying to automatically let the TS compiler output the TypeScript definitions (yellow block in the picture, generating `index.d.ts`), but I think we should generate ourselves during a build step, using a "generateTSDefinitions" function that we implement ourselves in the Dispatcher (yellow block in the picture). Shall we schedule a video call to think this through?

Typocomath architecture.svg

223 KiB

glen commented 2023-10-20 17:31:01 +00:00

Author

Owner

Copy Link

First, I just pushed a commit to dispatcher_refactor branch that demonstrates that indeed, the reflect macro can generate the generic interface needed to ultimately generate the type of the object that createDispatcher() will return. I swapped in the new macro in source files numbers/predicate.ts, Complex/arithmetic.ts, and Complex/predicate.ts. There is a slight hitch in that you now need to specify whether you are reflecting concrete or generic implementations, which means that sometimes you need to make two separate calls to the macro (see Complex/predicate.ts), but that doesn't seem like too big a deal to me...

I also tried out the $$typeMetadata! macro in Complex/all.ts; you can see the results by building and looking at build/Complex/all.js. It is a bit more structured, but we would still have to do a lot of parsing ourselves, so probably not worth switching. I'm not going to worry further about it unless I hear some interest from you. (For example, looking at the source code of ts-macros, we could likely refine what $$typeMetadata does considerably, so we could pursue making some PRs on ts-macros to get the info we need out of it in a more readily usable format. But personally I feel like that would be a distraction unless you really think we should pursue it.)

But second and more importantly, on the overall trajectory here:

As to your diagram, I agree that it represents one possible architecture. I guess whether it is the appropriate architecture depends on whether one conceives of the Dispatcher in typocomath as being (primarily) a JavaScript module or a TypeScript module. I guess in your diagram above, it's in the white box, and that conception is the white box is just JavaScript, not TypeScript (and in fact, I think if we are going with that architecture, we might want to have the source files corresponding to the white box be just .js files, not .ts).

On the other hand, if there's a conception that typocomath/mathjs 13.0 = mathts?? should be an intrinsically TypeScript-throughout enterprise, at least to the extent of using TypeScript everywhere that we can get away with (I don't doubt that in the innards of the dispatcher we will at least have to do a bunch of casting to very permissive types and/or type coercions), then the dispatcher module should be a TypeScript module, and as such createDispatcher() should accept a fully and precisely typed collection of implementations, and its return type should be the full and precise type of the operation bundle it generates. So in the case of creating the mathjs bundle with everything, the return type should include the information that it has a property add, and the type of that property will be a function overload (= intersection of function types) that can take two numbers and return a number, two Complex numbers and return a Complex number, and so on and etc.

I think you are right that the main pragmatic difference between these two conceptions consists in the yellow box. In the "intrinsically TypeScript" conception, the source file that produces the mathjs bundle will export the result of a createDispatcher() call, and if we have managed to fully type that function, then the bundle will inherently have the correct TypeScript type, and the .d.ts file that TypeScript produces anyway will be exactly the .d.ts file we want to publish. In the "white box is just JavaScript" conception, the source file that ultimately produces the mathjs bundle to publish will have to be just a JavaScript file since (more or less inherent to this conception) createDispatcher() won't be properly TypeScript-typed, and we will have to create some other mechanism to generate the .d.ts file that we will publish along with the mathjs file.

To me, writing the yellow box seems like a decent chunk of work, so we get two advantages if we manage to properly type createDispatcher(): (i) we avoid having to do that work, and (ii) we future-proof ourselves in that TypeScript continues to change noticeably from revision to revision (although the pace of change is slowing) and so if future .d.ts files need to be different in some way, well, presumably that future TypeScript will get the new .d.ts files right.

So I do think it's worth some investment of effort to get the typing of createDispatcher() fully detailed and correct.

As I see it, the five options (1), (2), (2b), (3), and (4) I outlined above represent a spectrum with respect to the idea of correctly typing createDispatcher()/having the Dispatcher module be typescript/having TypeScript generate the .d.ts file we want.

(1) corresponds to pure TypeScript. There, one must be able, within TypeScript, to express the type transformation from the specific detailed type of all of the implementation functions (with dependencies), to the type of the object createDispatcher() will produce. Based on my efforts and on StackOverflow responses, it may well be the case that with current TypeScript that's actually impossible, and I would be perfectly willing to give up on (1) altogether right now.

Options (2) and (2b) represent the possibility that with some additional type information, that can be gathered with some redundancy but modest overall effort, we can then compute the exact output type of createDispatcher(). So in this scenario, createDispatcher() would need another generic "helper" type parameter that can't be deduced and would have to be specified in its call. [It might be able to do at least some consistency checks between this helper type and the type of its parameter.] But when called with the proper helper type parameter, its output type would be totally detailed and correct. The difference between (2) and (2b) is just that in (2), we create that helper type with some extra stuff in the $reflect! macro, and in (2b), we create that helper type instead by stuffing all implementations into classes and then doing some class interface manipulations. (Note that it's possible that in 2b, we would not need to call the macro in each implementation file; it might be possible to call it just once for each type directory, in the the "all.ts" file. Not certain. Alternatively, if we make each file of implementations a class, it affords the opportunity to make $reflect! into a decorator macro, so we would be able to just insert it before each implementation as you had suggested earlier.) But in any case, these options remain solidly in the TypeScript world.

Option (3) says well, we're not going to get there with TypeScript, but we can get pretty close. It's basically represented in the current state of the dispatcher_refactor branch. We can generate not the correct type of the object returned by createDispatcher(), but the one in which the dependencies are still stuck in there. So we coerce the return type to that admittedly wrong type, compile to js, and then we take the generated .d.ts file and snip out all of the (dep: BLAH) => bits, leaving just the proper types of the operations in the generated bundle. It's admittedly pretty hacky, and fairly painful, to be deliberately mis-typing createDispatcher() to clean it up later in a text-manipulation step, but on the upside it's still TypeScript doing most of the interface generation work.

And finally option (4) is the "Dispatcher is just JavaScript" version although I agree you have clarified that there is still value in being able to write all the implementations in TypeScript and have TypeScript with the aid of macros be able to generate all of the necessary input information to the Dispatcher module. I think in this world, we would just make Dispatcher a javascript file, and as you say systematize that input information so that implementation specifications could just as well be coming from a pure JavaScript source, and presumably also allow all the dynamic manipulations to the dispatcher objects that we do in pocomath -- sure, they'd break the published typing of a specific bundle, but (a) if you don't care, you get the additional flexibility, and (b) if you do care, presumably that object would have a method to write out a new .d.ts file that you could use to interact with the modified object from TypeScript -- since we would have had to code up the full yellow box anyway, we might as well make it a method on the dispatcher object.

I'll close with my rankings of the effort level and conceptual attractiveness of the approaches. I am not going to list (2) and (2b) separately; I think they're pretty much the same on both these scales -- I think the main difference is that with (2b) we'll probably get slightly "prettier" source files, at the cost of the weirdness of "why are we stuffing all these arrow functions into classes?"

OK, so my guesses at effort level for us from least to most: (2/2b), (3), (4), (1) [because 1 is probably impossible]

Conceptual attractiveness to me: (1), (2/2b), (4), (3) [because 3 is clearly the most hacky]

Alright, I think I have fully braindumped on this topic; feel free to pick one or more of these directions for us to pursue (until the next roadblocks ;-), or we can meet as you suggest, whichever you prefer. Cheers!

First, I just pushed a commit to dispatcher_refactor branch that demonstrates that indeed, the reflect macro can generate the generic interface needed to ultimately generate the type of the object that createDispatcher() will return. I swapped in the new macro in source files numbers/predicate.ts, Complex/arithmetic.ts, and Complex/predicate.ts. There is a slight hitch in that you now need to specify whether you are reflecting concrete or generic implementations, which means that sometimes you need to make two separate calls to the macro (see Complex/predicate.ts), but that doesn't seem like too big a deal to me... I also tried out the `$$typeMetadata!` macro in Complex/all.ts; you can see the results by building and looking at build/Complex/all.js. It is a bit more structured, but we would still have to do a lot of parsing ourselves, so probably not worth switching. I'm not going to worry further about it unless I hear some interest from you. (For example, looking at the source code of ts-macros, we could likely refine what `$$typeMetadata` does considerably, so we could pursue making some PRs on ts-macros to get the info we need out of it in a more readily usable format. But personally I feel like that would be a distraction unless you really think we should pursue it.) But second and more importantly, on the overall trajectory here: As to your diagram, I agree that it represents one possible architecture. I guess whether it is the appropriate architecture depends on whether one conceives of the Dispatcher in typocomath as being (primarily) a JavaScript module or a TypeScript module. I guess in your diagram above, it's in the white box, and that conception is the white box is just JavaScript, not TypeScript (and in fact, I think if we are going with that architecture, we might want to have the source files corresponding to the white box be just .js files, not .ts). On the other hand, **if** there's a conception that typocomath/mathjs 13.0 = mathts?? should be an intrinsically TypeScript-throughout enterprise, at least to the extent of using TypeScript everywhere that we can get away with (I don't doubt that in the innards of the dispatcher we will at least have to do a bunch of casting to very permissive types and/or type coercions), then the dispatcher module should be a TypeScript module, and as such createDispatcher() should accept a fully and precisely typed collection of implementations, and its return type should be the full and precise type of the operation bundle it generates. So in the case of creating the mathjs bundle with everything, the return type should include the information that it has a property add, and the type of that property will be a function overload (= intersection of function types) that can take two numbers and return a number, two Complex numbers and return a Complex number, and so on and etc. I think you are right that the main pragmatic difference between these two conceptions consists in the yellow box. In the "intrinsically TypeScript" conception, the source file that produces the mathjs bundle will export the result of a createDispatcher() call, and if we have managed to fully type that function, then the bundle will inherently have the correct TypeScript type, and the .d.ts file that TypeScript produces anyway will be exactly the .d.ts file we want to publish. In the "white box is just JavaScript" conception, the source file that ultimately produces the mathjs bundle to publish will have to be just a JavaScript file since (more or less inherent to this conception) createDispatcher() won't be properly TypeScript-typed, and we will have to create some other mechanism to generate the .d.ts file that we will publish along with the mathjs file. To me, writing the yellow box seems like a decent chunk of work, so we get two advantages if we manage to properly type createDispatcher(): (i) we avoid having to do that work, and (ii) we future-proof ourselves in that TypeScript continues to change noticeably from revision to revision (although the pace of change is slowing) and so if future .d.ts files need to be different in some way, well, presumably that future TypeScript will get the new .d.ts files right. So I do think it's worth some investment of effort to get the typing of createDispatcher() fully detailed and correct. As I see it, the five options (1), (2), (2b), (3), and (4) I outlined above represent a spectrum with respect to the idea of correctly typing createDispatcher()/having the Dispatcher module be typescript/having TypeScript generate the .d.ts file we want. (1) corresponds to pure TypeScript. There, one must be able, within TypeScript, to express the type transformation from the specific detailed type of all of the implementation functions (with dependencies), to the type of the object createDispatcher() will produce. Based on my efforts and on StackOverflow responses, it may well be the case that with current TypeScript that's actually impossible, and I would be perfectly willing to give up on (1) altogether right now. Options (2) and (2b) represent the possibility that with some additional type information, that can be gathered with some redundancy but modest overall effort, we can then compute the exact output type of createDispatcher(). So in this scenario, createDispatcher() would need another generic "helper" type parameter that can't be deduced and would have to be specified in its call. [It might be able to do at least some consistency checks between this helper type and the type of its parameter.] But when called with the proper helper type parameter, its output type would be totally detailed and correct. The difference between (2) and (2b) is just that in (2), we create that helper type with some extra stuff in the `$reflect!` macro, and in (2b), we create that helper type instead by stuffing all implementations into classes and then doing some class interface manipulations. (Note that it's possible that in 2b, we would not need to call the macro in each implementation file; it might be possible to call it just once for each type directory, in the the "all.ts" file. Not certain. Alternatively, if we make each file of implementations a class, it affords the opportunity to make `$reflect!` into a decorator macro, so we would be able to just insert it before each implementation as you had suggested earlier.) But in any case, these options remain solidly in the TypeScript world. Option (3) says well, we're not going to get there with TypeScript, but we can get pretty close. It's basically represented in the current state of the dispatcher_refactor branch. We can generate not the correct type of the object returned by createDispatcher(), but the one in which the dependencies are still stuck in there. So we coerce the return type to that **admittedly wrong** type, compile to js, and then we take the generated .d.ts file and snip out all of the `(dep: BLAH) =>` bits, leaving just the proper types of the operations in the generated bundle. It's admittedly pretty hacky, and fairly painful, to be deliberately mis-typing createDispatcher() to clean it up later in a text-manipulation step, but on the upside it's still TypeScript doing most of the interface generation work. And finally option (4) is the "Dispatcher is just JavaScript" version although I agree you have clarified that there is still value in being able to write all the implementations in TypeScript and have TypeScript with the aid of macros be able to generate all of the necessary input information to the Dispatcher module. I think in this world, we would just make Dispatcher a javascript file, and as you say systematize that input information so that implementation specifications could just as well be coming from a pure JavaScript source, and presumably also allow all the dynamic manipulations to the dispatcher objects that we do in pocomath -- sure, they'd break the published typing of a specific bundle, but (a) if you don't care, you get the additional flexibility, and (b) if you do care, presumably that object would have a method to write out a new .d.ts file that you could use to interact with the modified object from TypeScript -- since we would have had to code up the full yellow box anyway, we might as well make it a method on the dispatcher object. I'll close with my rankings of the effort level and conceptual attractiveness of the approaches. I am not going to list (2) and (2b) separately; I think they're pretty much the same on both these scales -- I think the main difference is that with (2b) we'll probably get slightly "prettier" source files, at the cost of the weirdness of "why are we stuffing all these arrow functions into classes?" OK, so my guesses at effort level for us from least to most: (2/2b), (3), (4), (1) [because 1 is probably impossible] Conceptual attractiveness to me: (1), (2/2b), (4), (3) [because 3 is clearly the most hacky] Alright, I think I have fully braindumped on this topic; feel free to pick one or more of these directions for us to pursue (until the next roadblocks ;-), or we can meet as you suggest, whichever you prefer. Cheers!

josdejong commented 2023-10-23 18:41:40 +00:00

Collaborator

Copy Link

Having to specify whether to $reflect generic or not is indeed not ideal but acceptable at least for now. I agree on parking $$typeMetadata! for now, it's not a magic solution.

My idea is indeed that mathjs can function without TS, even when it is largely written in TS. I think it should be able to accept JS functions, like our current JS typed-functions.

To me, writing the yellow box seems like a decent chunk of work, so we get two advantages if we manage to properly type createDispatcher(): (i) we avoid having to do that work, and (ii) we future-proof ourselves in that TypeScript continues to change noticeably from revision to revision
[...]
So I do think it's worth some investment of effort to get the typing of createDispatcher() fully detailed and correct.

Yes, agree the ideal would be to write mathjs 100% in TS. I think though that that is not possible and we should find a good middleway. Option (3) is indeed hacky, I think we should not go that route.

I'm not entirely sure what you mean with option (2/2b), are you referring to the experiments that you did last week? Or was that the Option (1) approach? From these experiments I conclude though that it is very painful and probably impossible too. And there are some other issues that I see though and I think are not solvable with a pure TS approach:

1. One of the nice things of mathjs is that you can mix data types, like add(complex, number) when there is no explicit signature implemented for that. typed-function creates this at runtime and converts the input values to a matching signature like add(complex, complex). I think the type definitions exported by the new Dispatcher should contain all these "conversion signatures" into the index.d.ts, otherwise, users will not know that it is possible to do add(complex, number) (this is currently the case with the handcrafted type definitions too btw). I'm a bit afraid that the amount of signatures explodes, and I'm curious to see how bad that will get (or not). In any case, I think there is no way to figure out all these actual signatures at build time with pure TypeScript, except by running the Dispatcher for real and exporting all signatures from it in a build step.
2. In the Dispatcher, there will be rules for ordering signatures and data types and signatures with conversions. Will add(bignumber, number) return a number or a bignumber? That determines exactly which signature will be matched for specific input arguments (and what conversions will be done), and I don't think we can capture that logic in TS without using the Dispatcher itself. That will result in types that are mostly correct but not always.
3. Suppose that we do not want to support mixing data types like add(complex, number) or leave this unsupported in TS and only support it in the expression parser to keep things simple, we're not there, then we still get this working in the first place (and your experiments of last week show it is complicated). But please correct me if you still see promising options.

Therefore my feeling is that we best end up with something like option (4). The core of that approach would be to make a split between the types of (a) the input of the dispatcher (factory functions which are partly JS, partly TS+reflect), and (b) the output of the Dispatcher, which in principle are dynamic JS functions and a method to generate 100% accurate type definitions (which we can write to a file during a build step). I think this split will save us from most of the TypeScript headaches, and gives us all the freedom and control that we need.

Just to prevent confusion: I think we can (largely) write the code of the Dispatcher in TS and let it return a rough type definition like that it returns an object like { sqrt: function; add: function; ... }, not defining the exact signatures, only which functions are there and that they are a function.

Having to specify whether to `$reflect` generic or not is indeed not ideal but acceptable at least for now. I agree on parking `$$typeMetadata!` for now, it's not a magic solution. My idea is indeed that mathjs can function without TS, even when it is largely written in TS. I think it should be able to accept JS functions, like our current JS typed-functions. > To me, writing the yellow box seems like a decent chunk of work, so we get two advantages if we manage to properly type createDispatcher(): (i) we avoid having to do that work, and (ii) we future-proof ourselves in that TypeScript continues to change noticeably from revision to revision > [...] > So I do think it's worth some investment of effort to get the typing of createDispatcher() fully detailed and correct. Yes, agree the ideal would be to write mathjs 100% in TS. I think though that that is not possible and we should find a good middleway. Option (3) is indeed hacky, I think we should not go that route. I'm not entirely sure what you mean with option (2/2b), are you referring to the experiments that you did last week? Or was that the Option (1) approach? From these experiments I conclude though that it is very painful and probably impossible too. And there are some other issues that I see though and I think are not solvable with a pure TS approach: 1. One of the nice things of mathjs is that you can mix data types, like `add(complex, number)` when there is no explicit signature implemented for that. `typed-function` creates this at runtime and converts the input values to a matching signature like `add(complex, complex)`. I think the type definitions exported by the new Dispatcher should contain all these "conversion signatures" into the index.d.ts, otherwise, users will not know that it is possible to do `add(complex, number)` (this is currently the case with the handcrafted type definitions too btw). I'm a bit afraid that the amount of signatures explodes, and I'm curious to see how bad that will get (or not). In any case, I think there is no way to figure out all these actual signatures at build time with pure TypeScript, except by running the Dispatcher for real and exporting all signatures from it in a build step. 2. In the Dispatcher, there will be rules for ordering signatures and data types and signatures with conversions. Will `add(bignumber, number)` return a `number` or a `bignumber`? That determines exactly which signature will be matched for specific input arguments (and what conversions will be done), and I don't think we can capture that logic in TS without using the Dispatcher itself. That will result in types that are mostly correct but not always. 3. Suppose that we do not want to support mixing data types like `add(complex, number)` or leave this unsupported in TS and only support it in the expression parser to keep things simple, we're not there, then we still get this working in the first place (and your experiments of last week show it is complicated). But please correct me if you still see promising options. Therefore my feeling is that we best end up with something like option (4). The core of that approach would be to make a split between the types of (a) the input of the dispatcher (factory functions which are partly JS, partly TS+reflect), and (b) the output of the Dispatcher, which in principle are dynamic JS functions and a method to generate 100% accurate type definitions (which we can write to a file during a build step). I think this split will save us from most of the TypeScript headaches, and gives us all the freedom and control that we need. Just to prevent confusion: I think we can (largely) write the code of the Dispatcher in TS and let it return a rough type definition like that it returns an object like `{ sqrt: function; add: function; ... }`, not defining the exact signatures, only which functions are there and that they are a `function`.

glen commented 2023-10-23 19:46:22 +00:00

Author

Owner

Copy Link

Having to specify whether to $reflect generic or not is indeed not ideal but acceptable at least for now.

That's only needed for option (2), not any of the others, so will go away if we choose another route.

I agree on parking $$typeMetadata! for now, it's not a magic solution.

Check.

My idea is indeed that mathjs can function without TS, even when it is largely written in TS. I think it should be able to accept JS functions, like our current JS typed-functions.

Sure -- TS compiles to JS, so one can always use the resulting module from JS and just arrange the proper inputs, whatever format they end up as (e.g. the current "functions-with-reflectedType-property" format).

Yes, agree the ideal would be to write mathjs 100% in TS. I think though that that is not possible and we should find a good middleway. Option (3) is indeed hacky, I think we should not go that route.

Roger.

I'm not entirely sure what you mean with option (2/2b), are you referring to the experiments that you did last week?

Basically yes; these are options that attempt to generate the correct TypeScript type for the result of createDispatcher, staying within TypeScript, but using some additional type information beyond the type of the input functions to createDispatcher to generate the correct type. That additional type information has to be collected up in all of the implementation files and passed to createDispatcher as a generic type parameter.

Or was that the Option (1) approach?

No, that was attempting to do it solely from the type of the arguments to createDispatcher.

From these experiments I conclude though that it is very painful and probably impossible too.

My experiments/inquiries on Stack Overflow concluded that option (1) is probably currently impossible. Options (2/2b) still look feasible, although yes, they will involve extensive type manipulations as shown in the playground links I made.

And there are some other issues that I see though and I think are not solvable with a pure TS approach:

1. One of the nice things of mathjs is that you can mix data types, like add(complex, number) when there is no explicit signature implemented for that. typed-function creates this at runtime and converts the input values to a matching signature like add(complex, complex). I think the type definitions exported by the new Dispatcher should contain all these "conversion signatures" into the index.d.ts, otherwise, users will not know that it is possible to do add(complex, number) (this is currently the case with the handcrafted type definitions too btw). I'm a bit afraid that the amount of signatures explodes, and I'm curious to see how bad that will get (or not). In any case, I think there is no way to figure out all these actual signatures at build time with pure TypeScript, except by running the Dispatcher for real and exporting all signatures from it in a build step.

I completely agree that mathjs should still be able to add a Complex and a number and get a Complex by auto-conversion from number to Complex, and so that should be one of the signatures of the add method on the returned object. It is conceivable to me that the type computations needed to extend the signatures of all of the methods with the automatic conversions specified by the installed "Dispatcher-Types" are feasible within the TypeScript type system -- it's pretty powerful.

2. In the Dispatcher, there will be rules for ordering signatures and data types and signatures with conversions. Will add(bignumber, number) return a number or a bignumber? That determines exactly which signature will be matched for specific input arguments (and what conversions will be done), and I don't think we can capture that logic in TS without using the Dispatcher itself. That will result in types that are mostly correct but not always.

This is an extremely good point. What you are pointing out is that in the TypeScript type of the generated object, the overloads for each function property like .add must be in the correct order to reflect the type-matching algorithm of the Dispatcher. And I haven't a clue as to how to control that order in types generated by using complicated type computations in the TypeScript type language. So this point 2. might all by itself be a reason to decide on option (4).

3. Suppose that we do not want to support mixing data types like add(complex, number) or leave this unsupported in TS and only support it in the expression parser to keep things simple, we're not there, then we still get this working in the first place (and your experiments of last week show it is complicated). But please correct me if you still see promising options.

I think this is moot; personally I feel we need a solution that will automatically allow <T>add(Complex<T>, T) via the automatic conversion from T to Complex<T> via (t) => complex(t, zero(t)). To me that's one of the hallmarks of mathjs that makes it so comfortable for math folks to use.

Therefore my feeling is that we best end up with something like option (4). The core of that approach would be to make a split between the types of (a) the input of the dispatcher (factory functions which are partly JS, partly TS+reflect), and (b) the output of the Dispatcher, which in principle are dynamic JS functions and a method to generate 100% accurate type definitions (which we can write to a file during a build step). I think this split will save us from most of the TypeScript headaches, and gives us all the freedom and control that we need.

I think we are pretty harmonized on option (4). I was gung ho about trying to pursue (2/2b) for the sake of avoiding having to write code to write .d.ts files, which sounds about as pleasant as having teeth pulled, but your objection 2. above, which I had not yet considered, leaves me pretty cold as to our chances of success with (2/2b). So I am perfectly happy to resign ourselves to option (4) and writing the yellow box. But still happy to meet on Thursday to discuss.

Just to prevent confusion: I think we can (largely) write the code of the Dispatcher in TS and let it return a rough type definition like that it returns an object like { sqrt: function; add: function; ... }, not defining the exact signatures, only which functions are there and that they are a function.

Right, well, I think we should adopt a wait and see on that: if the types in Dispatcher become too permissive, then they cease to provide any benefit and just become a useless chore. So we can start writing in TypeScript and see if we think we're getting any meaningful type-safety, and if not, just ditch the type annotations and rename the file to a .js file.

> Having to specify whether to `$reflect` generic or not is indeed not ideal but acceptable at least for now. That's only needed for option (2), not any of the others, so will go away if we choose another route. > I agree on parking `$$typeMetadata!` for now, it's not a magic solution. Check. > My idea is indeed that mathjs can function without TS, even when it is largely written in TS. I think it should be able to accept JS functions, like our current JS typed-functions. Sure -- TS compiles to JS, so one can always use the resulting module from JS and just arrange the proper inputs, whatever format they end up as (e.g. the current "functions-with-reflectedType-property" format). > Yes, agree the ideal would be to write mathjs 100% in TS. I think though that that is not possible and we should find a good middleway. Option (3) is indeed hacky, I think we should not go that route. Roger. > I'm not entirely sure what you mean with option (2/2b), are you referring to the experiments that you did last week? Basically yes; these are options that attempt to generate the correct TypeScript type for the result of createDispatcher, staying within TypeScript, but using some additional type information beyond the type of the input functions to createDispatcher to generate the correct type. That additional type information has to be collected up in all of the implementation files and passed to createDispatcher as a generic type parameter. > Or was that the Option (1) approach? No, that was attempting to do it solely from the type of the arguments to createDispatcher. > From these experiments I conclude though that it is very painful and probably impossible too. My experiments/inquiries on Stack Overflow concluded that option (1) is probably currently impossible. Options (2/2b) still look feasible, although yes, they will involve extensive type manipulations as shown in the playground links I made. > And there are some other issues that I see though and I think are not solvable with a pure TS approach: > > 1. One of the nice things of mathjs is that you can mix data types, like `add(complex, number)` when there is no explicit signature implemented for that. `typed-function` creates this at runtime and converts the input values to a matching signature like `add(complex, complex)`. I think the type definitions exported by the new Dispatcher should contain all these "conversion signatures" into the index.d.ts, otherwise, users will not know that it is possible to do `add(complex, number)` (this is currently the case with the handcrafted type definitions too btw). I'm a bit afraid that the amount of signatures explodes, and I'm curious to see how bad that will get (or not). In any case, I think there is no way to figure out all these actual signatures at build time with pure TypeScript, except by running the Dispatcher for real and exporting all signatures from it in a build step. I completely agree that mathjs should still be able to add a Complex and a number and get a Complex by auto-conversion from number to Complex, and so that should be one of the signatures of the add method on the returned object. It is conceivable to me that the type computations needed to extend the signatures of all of the methods with the automatic conversions specified by the installed "Dispatcher-Types" are feasible within the TypeScript type system -- it's pretty powerful. > 2. In the Dispatcher, there will be rules for ordering signatures and data types and signatures with conversions. Will `add(bignumber, number)` return a `number` or a `bignumber`? That determines exactly which signature will be matched for specific input arguments (and what conversions will be done), and I don't think we can capture that logic in TS without using the Dispatcher itself. That will result in types that are mostly correct but not always. This is an extremely good point. What you are pointing out is that in the TypeScript type of the generated object, the overloads for each function property like `.add` must be in the correct order to reflect the type-matching algorithm of the Dispatcher. And I haven't a clue as to how to control that order in types generated by using complicated type computations in the TypeScript type language. So this point 2. might all by itself be a reason to decide on option (4). > 3. Suppose that we do not want to support mixing data types like `add(complex, number)` or leave this unsupported in TS and only support it in the expression parser to keep things simple, we're not there, then we still get this working in the first place (and your experiments of last week show it is complicated). But please correct me if you still see promising options. I think this is moot; personally I feel we _need_ a solution that will automatically allow `<T>add(Complex<T>, T)` via the automatic conversion from `T` to `Complex<T>` via `(t) => complex(t, zero(t))`. To me that's one of the hallmarks of mathjs that makes it so comfortable for math folks to use. > > Therefore my feeling is that we best end up with something like option (4). The core of that approach would be to make a split between the types of (a) the input of the dispatcher (factory functions which are partly JS, partly TS+reflect), and (b) the output of the Dispatcher, which in principle are dynamic JS functions and a method to generate 100% accurate type definitions (which we can write to a file during a build step). I think this split will save us from most of the TypeScript headaches, and gives us all the freedom and control that we need. I think we are pretty harmonized on option (4). I was gung ho about trying to pursue (2/2b) for the sake of avoiding having to write code to write .d.ts files, which sounds about as pleasant as having teeth pulled, but your objection 2. above, which I had not yet considered, leaves me pretty cold as to our chances of success with (2/2b). So I am perfectly happy to resign ourselves to option (4) and writing the yellow box. But still happy to meet on Thursday to discuss. > Just to prevent confusion: I think we can (largely) write the code of the Dispatcher in TS and let it return a rough type definition like that it returns an object like `{ sqrt: function; add: function; ... }`, not defining the exact signatures, only which functions are there and that they are a `function`. Right, well, I think we should adopt a wait and see on that: if the types in Dispatcher become too permissive, then they cease to provide any benefit and just become a useless chore. So we can start writing in TypeScript and see if we think we're getting any meaningful type-safety, and if not, just ditch the type annotations and rename the file to a .js file.

josdejong commented 2023-10-24 08:21:29 +00:00

Collaborator

Copy Link

I feel we need a solution that will automatically allow <T>add(Complex<T>, T) via the automatic conversion

Yes, totally agree!

Options (2/2b) still look feasible, although yes, they will involve extensive type manipulations as shown in the playground links I made.

It's the recurring theme with all our TS trials: it would be so nice if we can make it work, and it is so tempting to try make it work, but it is an endless uphill battle. And if we can make it work in TS now, we may get stuck in the future when we try to implement a new feature. I think we're at the point that we should accept a pragmatic solution.

I expect that extracting the TS types from the Dispatcher will be not that difficult though, the most complex part I think is the core of the dispatcher itself (which you already implemented in pocomath).

I think we are pretty harmonized on option (4).

Sounds like we have a plan 💪

So we can start writing in TypeScript and see if we think we're getting any meaningful type-safety, and if not, just ditch the type annotations and rename the file to a .js file.

Agree.

> I feel we need a solution that will automatically allow `<T>add(Complex<T>, T)` via the automatic conversion Yes, totally agree! > Options (2/2b) still look feasible, although yes, they will involve extensive type manipulations as shown in the playground links I made. It's the recurring theme with all our TS trials: it would be _so_ nice if we can make it work, and it is _so_ tempting to try make it work, but it is an endless uphill battle. And if we can make it work in TS now, we may get stuck in the future when we try to implement a new feature. I think we're at the point that we should accept a pragmatic solution. I expect that extracting the TS types from the Dispatcher will be not _that_ difficult though, the most complex part I think is the core of the dispatcher itself (which you already implemented in `pocomath`). > I think we are pretty harmonized on option (4). Sounds like we have a plan 💪 > So we can start writing in TypeScript and see if we think we're getting any meaningful type-safety, and if not, just ditch the type annotations and rename the file to a .js file. Agree.

glen commented 2023-10-26 19:02:37 +00:00

Author

Owner

Copy Link

OK, after our discussion today, we are definitely going to proceed with option (4). I will create a new issue with a list of subtasks to get to a working (but highly unoptimized) Dispatcher.

OK, after our discussion today, we are definitely going to proceed with option (4). I will create a new issue with a list of subtasks to get to a working (but highly unoptimized) Dispatcher.

👍 1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dispatcher_refactor

ts-macros-issues

approach6_infer_js_to_ts

experiment/typescript_plugin

ts-macros

type_registration

supply_runtime

json_schema

dktype

rtti

approach4.6

approach2.5

approach1.5

approach4.5

approach4_typealias

approach3_plain

signature_scheme

more_complex

easier_global_interface

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

refactor

Internal change needed for future work, without immediate behavioral change

  

wontfix

This won't be fixed

No Label

bug

duplicate

enhancement

help wanted

invalid

question

refactor

wontfix

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: glen/typocomath#19

No description provided.