How to specify dependencies non-redundantly? #1

Closed
opened 2022-12-02 16:06:24 +00:00 by glen · 32 comments
Owner

In glen/pocomath#55, we tentatively settled on giving an implementation to look something like

export const fn = {
   'subtract: infer': (dep: {
      add: (a: number, b: number) => number,
      negate: (a: number) => number
   }) => (a: number, b: number) => dep.add(a, dep.negate(b))
}

where subtract for number,number depends on add for number,number and negate for number. (Again, this is just a hypothetical, I am not going to implement subtract that way this time. But it could be any operation that depends on others.) Except for having to write all of those dep. prefixes, this looked pretty good.

However, in starting this prototype and actually trying to write a couple of things in TypeScript, I now notice the following problem: above we are mentioning the return type of negate on number. But this has been specified in the implementation of negate on number, so repeating it here is not DRY and leads to the maintenance problem that if we need to change the return type of negate (not plausible for that simple operation, but definitely could happen for other operations), then it will need to be changed at every use of negate, which seems like a real drawback.

Obviously to use an implementation of another operation with a specific signature, we have to mention the signature. But we need to find a way to specify that dependency without reiterating the return type, which is determined by the signature. And if we want nicely TypeScript-typed implementations like the above, it has to be possible for TypeScript at compilation time to know that return type, otherwise it won't be able to validate the call dep.add(a, dep.negate(b)).

In this case, it seems like we could just have TypeScript see the definitions of
add on number,number and negate on number:

import add_number_number from './add'
import negate_number from './negate'

export const fn = {
   'subtract: infer': (dep: {
      add: typeof add_number_number,
      negate: typeof negate_number
   }) => (a: number, b: number) => dep.add(a, dep.negate(b))
}

Ignoring for a moment the drawback of needing a name-mangling convention so that all of these implementations can be exported, I am not clear on how this would work for a hypothetical generic subtract, supposing that add and negate are not implemented as generics but rather with distinct functions for different signatures:

export const fn = {
   `subtract: infer`: <T>(dep: {
      add: typeof ??? // or (a: T, b: T) => ???
      negate: typeof ??? // or (a: T) => ??
    }) => (a: T, b: T) => dep.add(a, dep.negate(b))

Maybe we can make a generic type Returns<Op, Signature> in which Op is to be a string and Signature is to be a tuple type with a generic definition of none or something like that but specializations in each file with specific strings and tuples like

export typename Returns<'negate', [number]> = number

? Not sure if something like this would work. Just brainstorming. Suggestions are welcome.

In https://code.studioinfinity.org/glen/pocomath/issues/55, we tentatively settled on giving an implementation to look something like ``` export const fn = { 'subtract: infer': (dep: { add: (a: number, b: number) => number, negate: (a: number) => number }) => (a: number, b: number) => dep.add(a, dep.negate(b)) } ``` where subtract for `number,number` depends on `add` for `number,number` and `negate` for `number`. (Again, this is just a hypothetical, I am not going to implement subtract that way this time. But it could be any operation that depends on others.) Except for having to write all of those `dep.` prefixes, this _looked_ pretty good. However, in starting this prototype and actually trying to write a couple of things in TypeScript, I now notice the following problem: above we are mentioning the return type of `negate` on `number`. But this has been specified in the implementation of `negate` on `number`, so repeating it here is not DRY and leads to the maintenance problem that if we need to change the return type of `negate` (not plausible for that simple operation, but definitely could happen for other operations), then it will need to be changed at **every use** of negate, which seems like a real drawback. Obviously to use an implementation of another operation with a specific signature, we have to mention the signature. But we need to find a way to specify that dependency without reiterating the return type, which is determined by the signature. And if we want nicely TypeScript-typed implementations like the above, it has to be possible for TypeScript at compilation time to know that return type, otherwise it won't be able to validate the call `dep.add(a, dep.negate(b))`. In this case, it seems like we could just have TypeScript see the definitions of add on number,number and negate on number: ``` import add_number_number from './add' import negate_number from './negate' export const fn = { 'subtract: infer': (dep: { add: typeof add_number_number, negate: typeof negate_number }) => (a: number, b: number) => dep.add(a, dep.negate(b)) } ``` Ignoring for a moment the drawback of needing a name-mangling convention so that all of these implementations can be exported, I am not clear on how this would work for a hypothetical generic subtract, supposing that add and negate are not implemented as generics but rather with distinct functions for different signatures: ``` export const fn = { `subtract: infer`: <T>(dep: { add: typeof ??? // or (a: T, b: T) => ??? negate: typeof ??? // or (a: T) => ?? }) => (a: T, b: T) => dep.add(a, dep.negate(b)) ``` Maybe we can make a generic type `Returns<Op, Signature>` in which Op is to be a string and Signature is to be a tuple type with a generic definition of `none` or something like that but specializations in each file with specific strings and tuples like ``` export typename Returns<'negate', [number]> = number ``` ? Not sure if something like this would work. Just brainstorming. Suggestions are welcome.
Author
Owner

A possible drawback of the filling-in-a-generic-case-by-case suggestion above is that it seems like the declaration of every implementation for the dependency of a generic operation would have to be visible at the time the generic is defined? used? (Not sure which.) I worry this may break modularity...

A possible drawback of the filling-in-a-generic-case-by-case suggestion above is that it seems like the declaration of every implementation for the dependency of a generic operation would have to be visible at the time the generic is defined? used? (Not sure which.) I worry this may break modularity...
Author
Owner

Oh, TypeScript does not allow generics to be specialized for a particular instance the way C++ does. However, I am going to try to write a template that will search for implementations of a signature and give back the type of a dependency on that implementation. For example, if you want sqrt to depend on the complex function that takes two number arguments, you would write

export sqrt = (dep: ImplementationDependency<'complex', [number, number]>) => 
   n: number => {
      // code for sqrt goes here, may call dep.complex(m, n)
   }

The idea is that the ImplementationDependency<Name, ArgTuple> template will comb through all implementations looking for one that has a name in the Name type and that accepts arguments matching the ArgTuple type, and if it finds one return the type {complex: (...args: ArgTuple) => ReturnType<THE_MATCHED_FUNCTION>} which is the dependency object that would be passed in to give the meaning to dep.complex in the implementation. Then if there are multiple dependencies, their ImplementationDependency<blah, blee> types can be &ed together to get the exact needed type for dep.

Not 100% certain this will work, but at least it is a plausible approach. In particular, I have tested that the type of a function that takes [string, number] does match against example<T>(first: T, second: number), etc., so we have a hope of picking up both explicit and template implementations this way. However, I am not sure whether it will be able to get the correct return type in this generic case, i.e. in our particular instance the template is complex<T>(a: T, b:T): Complex<T> and we want the ImplementationDependency<'complex', [number, number]> to know that it is returning a Complex<number> not some generic Complex<T>. But we shall see.

I remain totally open to other approaches here...

Oh, TypeScript does not allow generics to be specialized for a particular instance the way C++ does. However, I am going to try to write a template that will search for implementations of a signature and give back the type of a dependency on that implementation. For example, if you want `sqrt` to depend on the `complex` function that takes two `number` arguments, you would write ``` export sqrt = (dep: ImplementationDependency<'complex', [number, number]>) => n: number => { // code for sqrt goes here, may call dep.complex(m, n) } ``` The idea is that the ImplementationDependency<Name, ArgTuple> template will comb through all implementations looking for one that has a name in the Name type and that accepts arguments matching the ArgTuple type, and if it finds one return the type `{complex: (...args: ArgTuple) => ReturnType<THE_MATCHED_FUNCTION>}` which is the dependency object that would be passed in to give the meaning to dep.complex in the implementation. Then if there are multiple dependencies, their ImplementationDependency<blah, blee> types can be `&`ed together to get the exact needed type for dep. Not 100% certain this will work, but at least it is a plausible approach. In particular, I have tested that the type of a function that takes `[string, number]` does match against `example<T>(first: T, second: number)`, etc., so we have a hope of picking up both explicit and template implementations this way. However, I am not sure whether it will be able to get the correct return type in this generic case, i.e. in our particular instance the template is `complex<T>(a: T, b:T): Complex<T>` and we want the `ImplementationDependency<'complex', [number, number]>` to know that it is returning a `Complex<number>` not some generic `Complex<T>`. But we shall see. I remain totally open to other approaches here...
Author
Owner

Hmm, yes, so far the return type in the generic case is posing a problem; I have asked on StackExchange: https://stackoverflow.com/questions/74679105/typescript-how-to-obtain-return-type-of-the-instance-of-a-generic-function-that

Hmm, yes, so far the return type in the generic case is posing a problem; I have asked on StackExchange: https://stackoverflow.com/questions/74679105/typescript-how-to-obtain-return-type-of-the-instance-of-a-generic-function-that
Author
Owner

Hmmm, well jcalz, who from all of the answers is pretty much the top expert on these sorts of things, says it can't be done. So I won't beat my head against that further. When I get a chance (can't work any more on this tday, and maybe not for a couple days), I will try just returning the generic function type as the result of ImplementationDependency<'complex', [number,number]> and seeing if that works... dunno.

Hmmm, well jcalz, who from all of the answers is pretty much the top expert on these sorts of things, says it can't be done. So I won't beat my head against that further. When I get a chance (can't work any more on this tday, and maybe not for a couple days), I will try just returning the generic function type as the result of `ImplementationDependency<'complex', [number,number]>` and seeing if that works... dunno.
Author
Owner

OK, the latest commit as of this writing has a first-pass solution for this issue.

Some items to note about this code organization/initial direction for the prototype:

  1. So far, the prototype is just to deal with specification and typing issues. There is no actual running math instance; the functions that install types and operation specifications are just dummies. The code of the specific implementations is intended to be "real," however.
  2. Every file with implementations (so far just numbers/type.ts, numbers/arithmetic.ts, and Complex/type.ts) just exports the types and operations that it wants, with the conventions that types have a _type suffix which is removed to get the name of the type, and operations can have no suffix, or any other suffix that starts with _, which would be removed to get the name. A corollary of this is that no operation can have an _ anywhere in its name (which is I think OK since mathjs has generally been using camelCase anyway). (And FYI, this means all of the exported names in one subdirectory have to be distinct, hence the complex_1 and complex_2 you will see in complex/type.ts -- the motivation for removing underscore suffixes to get the operation name. And the final rollup into the math object imports operations from different subdirectories into different subobjects, to avoid any name collisions in that step.)
  3. One benefit of this is that if you happen to need to use another operation on a specific signature and that implementation has no dependencies, you can just call it directly -- you don't need to use the dependency mechanism. See the implementation of sqrt in number/arithmetic.ts for an example of this; we just call unaryMinus.
  4. I've implemented a Dependency<'operation', [argtype, argtype, ...]> template type to look up the type of operation on the given argument types, so that it's easy to declare the dependencies and have TypeScript type them properly.
  5. You also just skip the dependencies if there are none, which seems like a plus, and if there are any I think they appear very distinctively with this template, so there should be no confusion.
  6. However, for this dependency type lookup to work, I need someplace for the TypeScript compiler to look these types up. To accomplish this, I've added a giant ImplementationTypes interface that they all need to get thrown into. And so then for any module that exports specific implementations, there needs to be a block at the end that augments the ImplementationTypes interface with all of those types, see the examples in number/type.ts and Complex/type.ts (there isn't such a block at the bottom of number/arithmetic.ts yet because nothing yet uses the functions from there).
  7. Those declare module "../core/Dispatcher" blocks are kind of ugly. They're redundant, too. Their only redeeming feature is that they are completely boilerplate and mechanical: they contain no new information and could easily be generated completely automatically.

So here are the two key points that I need feedback on:

A) how is the specification side shaping up from your point of view? Is this looking reasonable? I feel like we should get the specification of all of the operations to look just the way we will want it to be in the real mathjs 12 (or whatever we're calling it) before going very far. So is this looking like a reasonable start, or do you see needed course corrections already?

B) What do you want to do about those declare module "../core/Dispatcher" blocks? I can contemplate (i) leave them there and just have them in our code, and deal with the fact that when a new specific implementation is added or renamed, a corresponding line will have to be manually added or changed down in that section; (ii) write a little utility that will generate those blocks into separate files with some automatic name generation like Complex/type.ts produces Complex/type_impl_types.ts (the generated name will have to be referenced in TypeScript ///<reference ... comments, like the one at the top of number/arithmetic.ts), the utility will then run in the build process before tsc; (iii) beat my head against TypeScript more and see if I can come up with a way to look up those types without having to throw them into a common interface (and without having to import the corresponding files, otherwise there will be no way to create a number-only bundle of mathjs, if number/arithmetic has to actually import Complex/type.ts so that the code that is contingently used only if there are complex numbers will compile.) We could also consult some TypeScript experts on point (iii), especially if you know some; I am not too optimistic about my abilities to come up with anything better than the scheme in this proto-prototype. In particular, the fact that you could tell typescript to look up the types with a comment that did not lead to any actual importing in the object JavaScript code was a real boon in the current scheme.

For the record, I am personally OK with option (ii) above but would welcome any other thoughts you might have; I don't love any of the options.

One final comment: it seems clear to me at this point that the central engine that assembles multiple implementations for the same operation into functions that dispatch at run time on the types will have to be completely re-written for all of this to work like this and work in TypeScript. I.e., we won't be using typed-function as it stands, and it's sufficiently far from the current typed-function that it doesn't really make sense to do a typed-function v4 for mathjs 12. It basically needs to be re-written from scratch. So for now I've called it the Dispatcher, and will put it right into the core of the typocomath prototype. We can decide later whether it's worth publishing as a separate package like typed-function is. But what do you think about the name Dispatcher? I am totally open to any other name. I didn't call it anything so similar to typed-function because as far as I can see it will have to handle all of the different operations as an ensemble; they won't really be their own separate entities. So I did think of TypedUniverse or TypedEnsemble as possible names. But they seemed a little funny in a TypeScript concept, where the critical thing isn't the typing (as TypeScript already has that) but the runtime dispatch (which TypeScript very vehemently does not have). Hence Dispatcher, but if you don't like that name I am fine with changing it. Just let me know.

Thanks for your feedback!

OK, the [latest commit](https://code.studioinfinity.org/glen/typocomath/commit/ccc6153786aec6aac19dccec12a49a485b7687a6) as of this writing has a first-pass solution for this issue. Some items to note about this code organization/initial direction for the prototype: 1) So far, the prototype is just to deal with specification and typing issues. There is no actual running math instance; the functions that install types and operation specifications are just dummies. The code of the specific implementations is intended to be "real," however. 1) Every file with implementations (so far just numbers/type.ts, numbers/arithmetic.ts, and Complex/type.ts) just exports the types and operations that it wants, with the conventions that types have a `_type` suffix which is removed to get the name of the type, and operations can have no suffix, or any other suffix that starts with `_`, which would be removed to get the name. A corollary of this is that no operation can have an `_` anywhere in its name (which is I think OK since mathjs has generally been using camelCase anyway). (And FYI, this means all of the exported names in one subdirectory have to be distinct, hence the `complex_1` and `complex_2` you will see in complex/type.ts -- the motivation for removing underscore suffixes to get the operation name. And the final rollup into the math object imports operations from different subdirectories into different subobjects, to avoid any name collisions in that step.) 1) One benefit of this is that if you happen to need to use another operation on a specific signature and that implementation has no dependencies, you can just call it directly -- you don't need to use the dependency mechanism. See the implementation of sqrt in number/arithmetic.ts for an example of this; we just call unaryMinus. 1) I've implemented a `Dependency<'operation', [argtype, argtype, ...]>` template type to look up the type of `operation` on the given argument types, so that it's easy to declare the dependencies and have TypeScript type them properly. 1) You also just skip the dependencies if there are none, which seems like a plus, and if there are any I think they appear very distinctively with this template, so there should be no confusion. 1) However, for this dependency type lookup to work, I need someplace for the TypeScript compiler to look these types up. To accomplish this, I've added a giant ImplementationTypes interface that they all need to get thrown into. And so then for any module that exports specific implementations, there needs to be a block at the end that augments the ImplementationTypes interface with all of those types, see the examples in number/type.ts and Complex/type.ts (there isn't such a block at the bottom of number/arithmetic.ts yet because nothing yet _uses_ the functions from there). 1) Those `declare module "../core/Dispatcher"` blocks are kind of ugly. They're redundant, too. Their only redeeming feature is that they are completely boilerplate and mechanical: they contain _no_ new information and could easily be generated completely automatically. So here are the two key points that I need feedback on: A) how is the specification side shaping up from your point of view? Is this looking reasonable? I feel like we should get the specification of all of the operations to look just the way we will want it to be in the real mathjs 12 (or whatever we're calling it) before going very far. So is this looking like a reasonable start, or do you see needed course corrections already? B) What do you want to do about those `declare module "../core/Dispatcher"` blocks? I can contemplate (i) leave them there and just have them in our code, and deal with the fact that when a new specific implementation is added or renamed, a corresponding line will have to be manually added or changed down in that section; (ii) write a little utility that will generate those blocks into separate files with some automatic name generation like `Complex/type.ts` produces `Complex/type_impl_types.ts` (the generated name will have to be referenced in TypeScript `///<reference ...` comments, like the one at the top of `number/arithmetic.ts`), the utility will then run in the build process before `tsc`; (iii) beat my head against TypeScript more and see if I can come up with a way to look up those types without having to throw them into a common interface (and without having to import the corresponding files, otherwise there will be no way to create a number-only bundle of mathjs, if number/arithmetic has to actually _import_ Complex/type.ts so that the code that is contingently used only if there are complex numbers will compile.) We could also consult some TypeScript experts on point (iii), especially if you know some; I am not too optimistic about my abilities to come up with anything better than the scheme in this proto-prototype. In particular, the fact that you could tell typescript to look up the types with a **comment** that did not lead to any actual importing in the object JavaScript code was a real boon in the current scheme. For the record, I am personally OK with option (ii) above but would welcome any other thoughts you might have; I don't _love_ any of the options. One final comment: it seems clear to me at this point that the central engine that assembles multiple implementations for the same operation into functions that dispatch at run time on the types will have to be completely re-written for all of this to work like this and work in TypeScript. I.e., we won't be using typed-function as it stands, and it's sufficiently far from the current typed-function that it doesn't really make sense to do a typed-function v4 for mathjs 12. It basically needs to be re-written from scratch. So for now I've called it the Dispatcher, and will put it right into the core of the typocomath prototype. We can decide later whether it's worth publishing as a separate package like typed-function is. But what do you think about the name Dispatcher? I am totally open to any other name. I didn't call it anything so similar to typed-function because as far as I can see it will have to handle all of the different operations as an ensemble; they won't really be their own separate entities. So I did think of TypedUniverse or TypedEnsemble as possible names. But they seemed a little funny in a TypeScript concept, where the critical thing isn't the typing (as TypeScript already has that) but the runtime dispatch (which TypeScript very vehemently does not have). Hence Dispatcher, but if you don't like that name I am fine with changing it. Just let me know. Thanks for your feedback!
Collaborator

Thanks Glen for starting the prototype!

(A) I do not yet fully understand your approach, I suspect I'm two steps behind you and missing something. At this moment, I'm not yet understanding the need for the Dispatcher and Dependency<'operation', [argtype, argtype, ...]>, or the need for types like complex_1_Complex and complex_2_Complex.

To make sure we're on the same page: this is what I expect the prototype will be about:

  1. We go for a JS/TS hybrid solution for mathjs: utilize TS where possible and practical, stay away from complex TS stuff that requires expert knowledge. Figure out what is a good balance between JS/TS in the POC.
  2. Verify whether it possible to extend typed-function such that it can (runtime) generate accurate type definitions given an instantiated typed-function. Verify whether it possible to write these types into a file and publish the offline-generated types in an npm library that can be consumed by a TypeScript project.
  3. Verify wheether, in a typed-function, it possible to infer a signature from a TypeScript definition. This is about turning compile-time type information into a (runtime and/or compiletime) string with this type information that can be used in typed-function. Sort of like writing a macro in C. This is maybe possible by creating our own complier plugin, possibly a TypeScript plugin.

If I understand it correctly, the POC so far is focussing on modeling the syntax and typing of the types (like number, Complex, etc) and dependencies, and not yet adressing point (2) and (3).

In my head, writing functions could look something like the following:

export const add = {
  // support infer for simple TS interfaces
  // a TypeScript plugin will replace 'add:infer' with the actual TS signature during compilation
  'add:infer': () => (a: number, b: number) : number => a + b
}

export const sqrt = {
  // sqrt is maybe too complex to automatically infer from TS, 
  // so we just write out a conservative signature ourselves
  'sqrt(number) : number | Complex<number>': (dep: {
     config: Config,
     complex: (a: number, b: number) => Complex<number>
  }) => {
     if (dep.config.predictable || !dep.complex) {
        return (a: number) => isNaN(a) ? NaN : Math.sqrt(a)
     }
     return (a: number) => {
        if (isNaN(a)) return NaN
        if (a >= 0) return Math.sqrt(a)
        return dep.complex(0, Math.sqrt(unaryMinus(a)))
     }
  }
}

In order not to have to write out the signatures like (a: number, b: number) => Complex<number> again and again (not DRY like you say), we could extract the type, like:

// define (once)
type NumberNumberToComplex = (a: number, b: number) => Complex<number>

// implement (once)
const complex: NumberNumberToComplex = (a, b) => new Complex(a, b) 

// use as dependency (in multiple places)
export const sqrt = {
  'sqrt(number) : number | Complex<number>': (dep: {
     config: Config,
     complex: NumberNumberToComplex
  }) => {
     // ...
  }
}

How are you thinking about such an approach, would that be feasible? Can you explain a bit more about what advandages Dispatcher will bring and what the reasons for complex_1_Complex and complex_2_Complex are?

Thanks Glen for starting the prototype! (A) I do not yet fully understand your approach, I suspect I'm two steps behind you and missing something. At this moment, I'm not yet understanding the need for the `Dispatcher` and `Dependency<'operation', [argtype, argtype, ...]>`, or the need for types like `complex_1_Complex` and `complex_2_Complex`. To make sure we're on the same page: this is what I expect the prototype will be about: 1. We go for a JS/TS hybrid solution for `mathjs`: utilize TS where possible and practical, stay away from complex TS stuff that requires expert knowledge. Figure out what is a good balance between JS/TS in the POC. 2. Verify whether it possible to extend `typed-function` such that it can (runtime) generate accurate type definitions given an instantiated typed-function. Verify whether it possible to write these types into a file and publish the offline-generated types in an npm library that can be consumed by a TypeScript project. 3. Verify wheether, in a `typed-function`, it possible to infer a signature from a TypeScript definition. This is about turning compile-time type information into a (runtime and/or compiletime) string with this type information that can be used in typed-function. Sort of like writing a macro in C. This is maybe possible by creating our own complier plugin, possibly [a TypeScript plugin](https://www.typescriptlang.org/tsconfig#plugins). If I understand it correctly, the POC so far is focussing on modeling the syntax and typing of the types (like `number`, `Complex`, etc) and dependencies, and not yet adressing point (2) and (3). In my head, writing functions could look something like the following: ```ts export const add = { // support infer for simple TS interfaces // a TypeScript plugin will replace 'add:infer' with the actual TS signature during compilation 'add:infer': () => (a: number, b: number) : number => a + b } export const sqrt = { // sqrt is maybe too complex to automatically infer from TS, // so we just write out a conservative signature ourselves 'sqrt(number) : number | Complex<number>': (dep: { config: Config, complex: (a: number, b: number) => Complex<number> }) => { if (dep.config.predictable || !dep.complex) { return (a: number) => isNaN(a) ? NaN : Math.sqrt(a) } return (a: number) => { if (isNaN(a)) return NaN if (a >= 0) return Math.sqrt(a) return dep.complex(0, Math.sqrt(unaryMinus(a))) } } } ``` In order not to have to write out the signatures like `(a: number, b: number) => Complex<number>` again and again (not DRY like you say), we could extract the type, like: ```ts // define (once) type NumberNumberToComplex = (a: number, b: number) => Complex<number> // implement (once) const complex: NumberNumberToComplex = (a, b) => new Complex(a, b) // use as dependency (in multiple places) export const sqrt = { 'sqrt(number) : number | Complex<number>': (dep: { config: Config, complex: NumberNumberToComplex }) => { // ... } } ``` How are you thinking about such an approach, would that be feasible? Can you explain a bit more about what advandages `Dispatcher` will bring and what the reasons for `complex_1_Complex` and `complex_2_Complex` are?
Author
Owner

Sorry my explanations were not so clear.

We are indeed on the same page as to the outline of the approach on this prototype.

Dispatcher is just typed-function. But as far as I can see to make this version work as planned, typed-function will need to be completely rewritten from scratch so that it handles multiple operations referring to each other, template types, and extracting the types from TypeScript. I don't think it makes sense to try to evolve it from current typed-function. So I just called the new one Dispatcher to signal the rewrite, since in the TypeScript context it's not the types that are novel, it's the runtime dispatch.

Next comment will be about the Dependency mechanism.

Sorry my explanations were not so clear. We are indeed on the same page as to the outline of the approach on this prototype. Dispatcher is just typed-function. But as far as I can see to make this version work as planned, typed-function will need to be completely rewritten from scratch so that it handles multiple operations referring to each other, template types, and extracting the types from TypeScript. I don't think it makes sense to try to evolve it from current typed-function. So I just called the new one Dispatcher to signal the rewrite, since in the TypeScript context it's not the types that are novel, it's the runtime dispatch. Next comment will be about the Dependency mechanism.
Author
Owner

I am trying to make the definitions look just like the add and sqrt in your example, and if you look at the add in numbers/arithmetic you will see it is identical to yours, just without the extraneous first () => since it has no dependencies. Getting rid of that was a feature that I believe you wanted, and the new dependency mechanism enables dropping it, since Dispatcher can check if it gets a function that takes dependencies or just takes ordinary arguments.

But exactly as you say, the problem comes for the ones that do have dependencies. Since we want to write implementations in TypeScript, we need to type the dependency arguments. And how will those dependencies be identified? By the name of the operation needed and the types that it will take. It would be easy to write a TypeScript type transformer that would turn a string like complex(number, number) into the proper function type except for the return type of the supplied implementation of complex. That return type is specified at the original definition of complex(number, number), and we don't want to repeat it. It is true that we could export a type like ComplexNumberNumberImplementation in the file where complex(number,number) is defined, and then import type it in numbers/arithmetic, since import type does not create an actual import in the generated JavaScript.
That would totally work except for one hitch: there is no complex(number, number) defined, it's a template complex<T>(T, T). And we can't possibly write out all of the types of instantiations of this template. For example, a client of the library might install a type Foo and there definitely would not be a ComplexFooFooImplementation type in the complex/type file. Similarly, we wouldn't want within mathjs when we add a new type to have to hunt down every template and add another instantiation type.

So the solution is that we have to use TypeScript's type matching algorithm to look up which implementation of complex would be called on a signature of number,number. And that is exactly what Dependency<'complex', [number,number]> does. And the fact that the prototype compiles and runs (even though it does no computation) shows that lookup mechanism is working.

But to get it to work, there has to be someplace for it to do the lookup. The best thing I could come up with was a big interface with the types of all the implementations, where the keys are the operation names. Except since the operations can have lots of implementations, the keys can't be exactly the operation names, because they would name-clash and coalesce into an overload, and TypeScript is very poor at doing type inference with overloads, as well documented online. Hence the suffixes with the easily parseable _ separator. And the suffixes need to have enough in them to be sure to be unique, hence the final _[DIRECTORY] piece and the arbitrary _1 and _2 since there happen to be two implementations of complex in the Complex directory. Those could have been anything, like complex_unary and complex_binary might be clearer. I just used _1 and _2 at the moment to emphasize their arbitrariness.

Hope that clarifies the prototype a bit and the motivations for the choices so far. Thoughts?

I am trying to make the definitions look just like the add and sqrt in your example, and if you look at the add in numbers/arithmetic you will see it is identical to yours, just without the extraneous first `() =>` since it has no dependencies. Getting rid of that was a feature that I believe you wanted, and the new dependency mechanism enables dropping it, since Dispatcher can check if it gets a function that takes dependencies or just takes ordinary arguments. But exactly as you say, the problem comes for the ones that do have dependencies. Since we want to write implementations in TypeScript, we need to type the dependency arguments. And how will those dependencies be identified? By the name of the operation needed and the types that it will take. It would be easy to write a TypeScript type transformer that would turn a string like `complex(number, number)` into the proper function type _except_ for the return type of the supplied implementation of complex. That return type is specified at the original definition of complex(number, number), and we don't want to repeat it. It is true that we could export a type like `ComplexNumberNumberImplementation` in the file where `complex(number,number)` is defined, and then `import type` it in numbers/arithmetic, since `import type` does _not_ create an actual import in the generated JavaScript. That would totally work except for one hitch: there is no `complex(number, number)` defined, it's a template `complex<T>(T, T)`. And we can't possibly write out all of the types of instantiations of this template. For example, a client of the library might install a type `Foo` and there definitely would not be a ComplexFooFooImplementation type in the complex/type file. Similarly, we wouldn't want within mathjs when we add a new type to have to hunt down every template and add another instantiation type. So the solution is that we have to use TypeScript's type matching algorithm to look up which implementation of complex would be called on a signature of number,number. And that is exactly what `Dependency<'complex', [number,number]>` does. And the fact that the prototype compiles and runs (even though it does no computation) shows that lookup mechanism is working. But to get it to work, there has to be someplace for it to do the lookup. The best thing I could come up with was a big interface with the types of all the implementations, where the keys are the operation names. Except since the operations can have lots of implementations, the keys can't be exactly the operation names, because they would name-clash and coalesce into an overload, and TypeScript is very poor at doing type inference with overloads, as well documented online. Hence the suffixes with the easily parseable `_` separator. And the suffixes need to have enough in them to be sure to be unique, hence the final `_[DIRECTORY]` piece and the arbitrary `_1` and `_2` since there happen to be two implementations of `complex` in the `Complex` directory. Those could have been anything, like `complex_unary` and `complex_binary` might be clearer. I just used `_1` and `_2` at the moment to emphasize their arbitrariness. Hope that clarifies the prototype a bit and the motivations for the choices so far. Thoughts?
Author
Owner

Verify wheether, in a typed-function, it possible to infer a signature from a TypeScript definition. This is about turning compile-time type information into a (runtime and/or compiletime) string with this type information that can be used in typed-function. Sort of like writing a macro in C. This is maybe possible by creating our own complier plugin, possibly a TypeScript plugin.

I am planning to use the published typescript-rtti package for this.

> Verify wheether, in a typed-function, it possible to infer a signature from a TypeScript definition. This is about turning compile-time type information into a (runtime and/or compiletime) string with this type information that can be used in typed-function. Sort of like writing a macro in C. This is maybe possible by creating our own complier plugin, possibly a TypeScript plugin. I am planning to use the published [typescript-rtti](https://www.npmjs.com/package/typescript-rtti) package for this.
Author
Owner

And one last comment for now: for sqrt you write

'sqrt(number) : number | Complex<number>': (dep: {

whereas the prototype in numbers/arithmetic currently has

export const sqrt =
   (dep: ... stuff

with no explicit type. That's because the return type of sqrt differs depending on the config, and we want typed-function aka Dispatcher to know that. If we explicitly type it as returning number|Complex<number> then it won't work in the number-only bundle of mathjs; it won't be able to make sense of the Complex<number> type. So this way there's at least a fighting chance both TypeScript and Dispatcher will be able to perform correct type inference; we shall see if it all works as the prototype evolves. I am a bit worried on this point whether future uses of sqrt will type in TypeScript correctly, since at compile time it can't know the config and so sqrt will be ambiguous between returning a number and returning a number|Complex<number>, i.e. it will have as a type the union of two function types, which TypeScript isn't great at handling. But we shall see; I am not worrying about this particular point until it comes up, which it definitely will in implementing the polynomial root finder.

And one last comment for now: for sqrt you write ``` 'sqrt(number) : number | Complex<number>': (dep: { ``` whereas the prototype in numbers/arithmetic currently has ``` export const sqrt = (dep: ... stuff ``` with no explicit type. That's because the return type of sqrt differs depending on the config, and we want typed-function aka Dispatcher to know that. If we explicitly type it as returning `number|Complex<number>` then it won't work in the number-only bundle of mathjs; it won't be able to make sense of the `Complex<number>` type. So this way there's at least a fighting chance both TypeScript and Dispatcher will be able to perform correct type inference; we shall see if it all works as the prototype evolves. I _am_ a bit worried on this point whether future _uses_ of sqrt will type in TypeScript correctly, since at compile time it can't know the config and so sqrt will be ambiguous between returning a number and returning a `number|Complex<number>`, i.e. it will have as a type the union of two function types, which TypeScript isn't great at handling. But we shall see; I am not worrying about this particular point until it comes up, which it definitely will in implementing the polynomial root finder.
Collaborator

I am planning to use the published typescript-rtti package for this.

o wow, that is exactly what we need 😎

So, trying to understand the complexity around template types: here is where I come from. In pocomath, you implemented generic types (template types). I guess it makes sense to move this logic into typed-function, but in any case, this logic has been implemented in JS and works. Now, we try to write a function with as much TS as possible. Concrete examples help me here, so let's look at the function invert from pocomath:

// js variant of invert from pocomath
export const invertJs = {
   'Complex<T>': ({
      T,
      'conjugate(Complex<T>)': conj,
      'absquare(Complex<T>)': asq,
      'complex(T,T)': cplx,
      'divide(T,T)': div
   }) => Returns(`Complex<${T}>`, z => {
      const c = conj(z)
      const d = asq(z)
      return cplx(div(c.re, d), div(c.im, d))
   })
}

We can write this in TypeScript like:

interface Complex<T> {
   re: T
   im: T
}

// ts variant of invert
export const invertTs = {
   'invert: infer': function <T>(deps: {
      conjugate: (a: Complex<T>) => Complex<T>,
      absquare: (a: Complex<T>) => T,
      complex: (re: T, im: T) => Complex<T>,
      divide: (a: T, b: T) => T
   }) {
      return function (z: Complex<T>): Complex<T> {
         const c = deps.conjugate(z)
         const d = deps.absquare(z)
         return deps.complex(deps.divide(c.re, d), deps.divide(c.im, d))
      }
   }
}

We need a compiler step to rewrite the TS code into the JS code from above again, but this looks quite straight forward to me. I'm still missing where the complexity around template types and the need for looking up implementations that you describe above will pop in in practical cases. Can you try to explain?

Maybe the difference is that I'm just fine with writing out these signatures repeatedly, and you would like to see a 100% DRY solution? We could extract (a: Complex<T>) => Complex<T> into a type ComplexTToComplexT for example to reduce repetition, but besides that, I do not see a problem with this approach.

> I am planning to use the published [typescript-rtti](https://www.npmjs.com/package/typescript-rtti) package for this. o wow, that is _exactly_ what we need 😎 So, trying to understand the complexity around template types: here is where I come from. In pocomath, you implemented generic types (template types). I guess it makes sense to move this logic into typed-function, but in any case, this logic has been implemented in JS and works. Now, we try to write a function with as much TS as possible. Concrete examples help me here, so let's look at [the function `invert` from pocomath](https://code.studioinfinity.org/glen/pocomath/src/branch/main/src/complex/invert.mjs): ```js // js variant of invert from pocomath export const invertJs = { 'Complex<T>': ({ T, 'conjugate(Complex<T>)': conj, 'absquare(Complex<T>)': asq, 'complex(T,T)': cplx, 'divide(T,T)': div }) => Returns(`Complex<${T}>`, z => { const c = conj(z) const d = asq(z) return cplx(div(c.re, d), div(c.im, d)) }) } ``` We can write this in TypeScript like: ```ts interface Complex<T> { re: T im: T } // ts variant of invert export const invertTs = { 'invert: infer': function <T>(deps: { conjugate: (a: Complex<T>) => Complex<T>, absquare: (a: Complex<T>) => T, complex: (re: T, im: T) => Complex<T>, divide: (a: T, b: T) => T }) { return function (z: Complex<T>): Complex<T> { const c = deps.conjugate(z) const d = deps.absquare(z) return deps.complex(deps.divide(c.re, d), deps.divide(c.im, d)) } } } ``` We need a compiler step to rewrite the TS code into the JS code from above again, but this looks quite straight forward to me. I'm still missing where the complexity around template types and the need for looking up implementations that you describe above will pop in in practical cases. Can you try to explain? Maybe the difference is that I'm just fine with writing out these signatures repeatedly, and you would like to see a 100% DRY solution? We could extract `(a: Complex<T>) => Complex<T>` into a type `ComplexTToComplexT` for example to reduce repetition, but besides that, I do not see a problem with this approach.
Author
Owner

For convenience, let's use the unary form of complex as the example. You just wrote a generic implementation, let's say it used the unary complex instead of the binary. So it needs to call complex on an argument of type T, which should return Complex<T> so that needs to be a generic function but fine in the file that defines complex we define a type complexOfT as <T>(a:T)=>Complex<T> and declare the dependency here to be of type complexOfT.

Now in sqrt, we just want complex of a concrete number argument, so we need to add a properly defined complexOfNumber type so that sqrt will be able to declare its dependency. But now in the bigint sqrt, we will need complexOfBigint. So you see, there is an explosion of type declarations we need, for all combinations of argument types that clients of the operation might want. And then a person extending mathjs adds a type Foo and wants to write a function that depends on the unary complex function with a Foo argument. mathjs is perfectly capable of doing that, but there is of course no type complexOfFoo defined anywhere for the person to declare their dependency on that function.

So the obstruction is without TypeScript doing the matching to find the implementation that would be called on a given signature, there is an explosion of type declarations and a real obstacle to extensibility. In other words, we would be manually typing all of the instantiations of a generic implementation when TypeScript is perfectly capable of doing the matching/typing for us.

And you can't say that the clients all just use the generic type complexOfT because they shouldn't have to know whether an operation is defined as a generic or just as a collection of individual implementations one for each type. (And because that implementation choice might change for an individual operation.) Both cases occur, and mixed cases where there is a generic overriden for some type by a specific implementation. To avoid entanglement of the code, therefore, the client needs some way to look up the proper dependency signature just from the operation name "op" and the types they want to call it with, say number and string, whether that be using a conventional identifier for a type like opOfNumberString or a template like Dependency<'op',[number,string]>

But again, as far as I have been able to come up with, TypeScript can only only do the lookup if we have a specific place to look things up. Although it just now occurs to me that maybe we could have a different interface object for each operation rather than one big one; that might simplify the implementation files a bit. Then the Dependency template might change to something like Dependency<complexImplementation, [number]> where complexImplementation is that interface that is added to each time someone adds an implementation for complex. (Maybe we should use a shorter conventional name and one that starts with a capital letter, like Icomplex.)
Should I give that a try? It may require adding a module or modules that just define empty interfaces for each operation, so there is something to extend...

For convenience, let's use the unary form of complex as the example. You just wrote a generic implementation, let's say it used the unary complex instead of the binary. So it needs to call complex on an argument of type T, which should return `Complex<T>` so that needs to be a generic function but fine in the file that defines complex we define a type complexOfT as `<T>(a:T)=>Complex<T>` and declare the dependency here to be of type complexOfT. Now in sqrt, we just want complex of a concrete number argument, so we need to add a properly defined complexOfNumber type so that sqrt will be able to declare its dependency. But now in the bigint sqrt, we will need complexOfBigint. So you see, there is an explosion of type declarations we need, for all combinations of argument types that clients of the operation might want. And then a person extending mathjs adds a type Foo and wants to write a function that depends on the unary complex function with a Foo argument. mathjs is perfectly capable of doing that, but there is of course no type complexOfFoo defined anywhere for the person to declare their dependency on that function. So the obstruction is without TypeScript doing the matching to find the implementation that would be called on a given signature, there is an explosion of type declarations and a real obstacle to extensibility. In other words, we would be manually typing all of the instantiations of a generic implementation when TypeScript is perfectly capable of doing the matching/typing for us. And you can't say that the clients all just use the generic type complexOfT because they shouldn't have to know whether an operation is defined as a generic or just as a collection of individual implementations one for each type. (And because that implementation choice might change for an individual operation.) Both cases occur, and mixed cases where there is a generic overriden for some type by a specific implementation. To avoid entanglement of the code, therefore, the client needs some way to look up the proper dependency signature just from the operation name "op" and the types they want to call it with, say number and string, whether that be using a conventional identifier for a type like `opOfNumberString` or a template like `Dependency<'op',[number,string]>` But again, as far as I have been able to come up with, TypeScript can only only do the lookup if we have a specific place to look things up. Although it just now occurs to me that maybe we could have a different interface object for each operation rather than one big one; that might simplify the implementation files a bit. Then the Dependency template might change to something like `Dependency<complexImplementation, [number]>` where complexImplementation is that interface that is added to each time someone adds an implementation for complex. (Maybe we should use a shorter conventional name and one that starts with a capital letter, like Icomplex.) Should I give that a try? It may require adding a module or modules that just define empty interfaces for each operation, so there is something to extend...
Author
Owner

P.S. There is another reason why I don't think just writing out all dependency types explicitly will work. Take absquare; its return type for number is number, but for Complex<number> is also number. But suppose you are writing a generic that needs to call absquare on an arbitrary type. There is nothing you can write explicitly: absquare: (a:T) => T is wrong when T is Complex<number> But absquare: (a:Complex<T>) => T would limit your generic to Complex types and is wrong anyway on quaternions = Complex<Complex<number>> where absquare still returns number. So it seems that you need a generic like Dependency<'absquare', [T]> to look up based on the type T what the proper dependency type should be.

so that goes beyond just the tedium and non-DRYness of always writing add: (a:number, b:number) => number to actual operational adequacy.

P.S. There is another reason why I don't think just writing out all dependency types explicitly will work. Take absquare; its return type for number is number, but for `Complex<number>` is also number. But suppose you are writing a generic that needs to call absquare on an arbitrary type. There is nothing you can write explicitly: `absquare: (a:T) => T` is wrong when T is `Complex<number>` But `absquare: (a:Complex<T>) => T` would limit your generic to Complex types and is wrong anyway on quaternions = `Complex<Complex<number>>` where absquare still returns number. So it seems that you need a generic like `Dependency<'absquare', [T]>` to look up based on the type T what the proper dependency type should be. so that goes beyond just the tedium and non-DRYness of always writing `add: (a:number, b:number) => number` to actual operational adequacy.
Collaborator

Thanks, I better understand your concerns now.

So you see, there is an explosion of type declarations we need, for all combinations of argument types that clients of the operation might want.

I'm not sure. I think the amount of type declarations in typocomath is just the same as in pocomath, and only has a different syntax, right? Or do you mean something else? There will be just one interface for every implementation, that doesn't sound weird to me, it is just the nature of wanting types I guess.

To me, typing either Dependency<'complex', [number, number]> or complex: (re: T, im: T) => Complex<T> is more or less the same: a different syntax, but you have to type about the same amount of code. The main difference is that the first approach automatically detects a return type, and ensures there is an implemenation right? That looks handy but I'm not sure if this is something we should want.

In general, when you have dependency injection, the whole point is that the function at hand does not know if and which implementations there are for each of its dependencies. It is unaware of that. It only specifies interfaces that the dependencies should adhere to. Now, of course, it will be helpful to have reuseable interfaces.

So, looking at the example I gave earlier, I think it is essential that the code of sqrt() is not coupled with the implementation of complex() and vice versa. They can both be use the shared interface NumberNumberToComplex (DRY), but they must not know/require each others existence. To have a matching interface, both the arguments and the return type should match.

// define interface (to be used both by the "producer" and "consumer")
type NumberNumberToComplex = (a: number, b: number) => Complex<number>

// implementation
// the "producer", complex(), uses the interface NumberNumberToComplex and does not know about sqrt
const complex: NumberNumberToComplex = (a, b) => new Complex(a, b) 

// function using a dependency
// the "consumer", sqrt(), uses the interface NumberNumberToComplex and does not know about complex
export const sqrt = {
  'sqrt: infer': (dep: {
     config: Config,
     complex: NumberNumberToComplex
  }) => {
     // ...
  }
}

I do not fully understand your example with abssquare. Can you write out a minimal TS example demonstrating the problem?

Still, I do not see why my examples of 1#issuecomment-835 would not work or why it would result in an unwieldy amount of interfaces or non-DRY code or anything.

Let's plan for a video call to talk this through, I think that will work better (I see I start repeating myself 😅).

Thanks, I better understand your concerns now. > So you see, there is an explosion of type declarations we need, for all combinations of argument types that clients of the operation might want. I'm not sure. I think the amount of type declarations in typocomath is just the same as in pocomath, and only has a different syntax, right? Or do you mean something else? There will be just one interface for every implementation, that doesn't sound weird to me, it is just the nature of wanting types I guess. To me, typing either `Dependency<'complex', [number, number]>` or `complex: (re: T, im: T) => Complex<T>` is more or less the same: a different syntax, but you have to type about the same amount of code. The main difference is that the first approach automatically detects a return type, and ensures there is an implemenation right? That looks handy but I'm not sure if this is something we should want. In general, when you have dependency injection, the whole point is that the function at hand does _not_ know _if_ and _which_ implementations there are for each of its dependencies. It is unaware of that. It only specifies interfaces that the dependencies should adhere to. Now, of course, it will be helpful to have reuseable interfaces. So, looking at the example I gave earlier, I think it is essential that the code of `sqrt()` is _not_ coupled with the _implementation_ of `complex()` and vice versa. They can both be use the shared interface `NumberNumberToComplex` (DRY), but they must not know/require each others existence. To have a matching interface, both the arguments and the return type should match. ```ts // define interface (to be used both by the "producer" and "consumer") type NumberNumberToComplex = (a: number, b: number) => Complex<number> // implementation // the "producer", complex(), uses the interface NumberNumberToComplex and does not know about sqrt const complex: NumberNumberToComplex = (a, b) => new Complex(a, b) // function using a dependency // the "consumer", sqrt(), uses the interface NumberNumberToComplex and does not know about complex export const sqrt = { 'sqrt: infer': (dep: { config: Config, complex: NumberNumberToComplex }) => { // ... } } ``` I do not fully understand your example with `abssquare`. Can you write out a minimal TS example demonstrating the problem? Still, I do not see why my examples of [1#issuecomment-835](https://code.studioinfinity.org/glen/typocomath/issues/1#issuecomment-835) would not work or why it would result in an unwieldy amount of interfaces or non-DRY code or anything. Let's plan for a video call to talk this through, I think that will work better (I see I start repeating myself 😅).
Author
Owner

There will be just one interface for every implementation, that doesn't sound weird to me, it is just the nature of wanting types I guess.

No, that's precisely the problem: for an implementation defined as a generic, there needs to be one typing to cover the places it is used as a generic, and another typing for each place it is used not as a generic, but as a specific instantiation for each different type that it is used as -- precisely to avoid that code entanglement, because the using code should not need to know that the operation was implemented as a generic or as a specific version for that typing, it should just ask for the operation on the types it needs. But now there is no way to know ahead of time what all types the generic is going to be needed for, especially because it might be for types that don't even exist in mathjs at the time the generic is written (such as if a client extends mathjs with a new type). That's why we have to use TypeScript's type matching algorithm, since it knows that a generic can handle a specific type that's asked for, regardless of what type that is.

> There will be just one interface for every implementation, that doesn't sound weird to me, it is just the nature of wanting types I guess. No, that's precisely the problem: for an implementation defined as a generic, there needs to be one typing to cover the places it is used as a generic, and another typing for each place it is used not as a generic, but as a specific instantiation for each different type that it is used as -- precisely to avoid that code entanglement, because the using code should not need to know that the operation was implemented as a generic or as a specific version for that typing, it should just ask for the operation on the types it needs. But now there is no way to know ahead of time what all types the generic is going to be needed for, especially because it might be for types that don't even exist in mathjs at the time the generic is written (such as if a client extends mathjs with a new type). That's why we have to use TypeScript's type matching algorithm, since it knows that a generic can handle a specific type that's asked for, regardless of what type that is.
Author
Owner

I do not fully understand your example with abssquare. Can you write out a minimal TS example demonstrating the problem?

This is the converse problem to the previous one. abssquare as it stands in pocomath is a mix of specifics for types like number and bigint and Fraction, etc., and generics for Complex<T>. And the return type is complicated: for a bigint it is bigint, for a number it is number, for a Complex<number> it is number, for a Complex<Complex<number>> it is number, etc. But now now consider the definition of abs: it should be just sqrt(absquare(x)). That is a fully generic definition, if it types properly it will work for any type for x. But how are we going to declare the dependency on absquare?

export const abs = (dep: {
  absquare: <T>(x: T) => ???,
  ...

Even if we wanted to go to a scheme in which we wrote out type definitions for each operation called on certain argument types, like

type absquareComplexNumber = (z: Complex<number>) => number

here we would want the typing of the generic

type absquareT = <T>(x: T) => ???

But there is no simple type expression in T that tells you the type of absquare(t) when t is of type T. That's why we need to define a type operator that will cause TypeScript, for any specific type T that it actually encounters, to look up the type of the absquare operation on an argument of type T.

To sum up these two posts: there are typing difficulties with specific uses of a generic implementation, and typing difficulties with generic uses of an operation that has different specific (or a mix of generic and specific) implementations, both of which can be addressed by providing a type operator that looks up the proper type of an implementation, as opposed to simply trying to define type constants for each implementation.

> I do not fully understand your example with abssquare. Can you write out a minimal TS example demonstrating the problem? This is the converse problem to the previous one. abssquare as it stands in pocomath is a mix of specifics for types like number and bigint and Fraction, etc., and generics for `Complex<T>`. And the return type is complicated: for a bigint it is bigint, for a number it is number, for a `Complex<number>` it is number, for a `Complex<Complex<number>>` it is number, etc. But now now consider the definition of abs: it should be just `sqrt(absquare(x))`. That is a fully generic definition, if it types properly it will work for any type for `x`. But how are we going to declare the dependency on absquare? ``` export const abs = (dep: { absquare: <T>(x: T) => ???, ... ``` Even if we wanted to go to a scheme in which we wrote out type definitions for each operation called on certain argument types, like `type absquareComplexNumber = (z: Complex<number>) => number` here we would want the typing of the generic `type absquareT = <T>(x: T) => ???` But there is no simple type expression in T that tells you the type of absquare(t) when t is of type T. That's why we need to define a type operator that will cause TypeScript, for any specific type T that it actually encounters, to look up the type of the absquare operation on an argument of type T. To sum up these two posts: there are typing difficulties with specific uses of a generic implementation, and typing difficulties with generic uses of an operation that has different specific (or a mix of generic and specific) implementations, both of which can be addressed by providing a type operator that looks up the proper type of an implementation, as opposed to simply trying to define type constants for each implementation.
Author
Owner

Just to summarize a bit from our chat: The plan is currently for me to flesh out a bit more of the types and implementations making sure that they all still compile (without worrying about getting them to do something). Then you will try writing the same specifications in a more "straightforward" notation and see if you encounter difficulties/see how the two versions compare.

And actually, we didn't talk about this, byt I will try two branches on my end: one just like now where you write Dependency<'foo', [number, string]> where all of the implementation types are thrown into one giant interface, and another Dependency<fooImpl, [number, string]> where there is a different interface for each operation foo, called fooImpl by convention. The reason for the second version is that I have a thought that maybe by avoiding a single giant interface I can avoid some of the complexity of those "declare module" sections or possibly eliminate them altogether. I'll post here when there's new stuff working to look at.

Just to summarize a bit from our chat: The plan is currently for me to flesh out a bit more of the types and implementations making sure that they all still compile (without worrying about getting them to do something). Then you will try writing the same specifications in a more "straightforward" notation and see if you encounter difficulties/see how the two versions compare. And actually, we didn't talk about this, byt I will try two branches on my end: one just like now where you write `Dependency<'foo', [number, string]>` where all of the implementation types are thrown into one giant interface, and another `Dependency<fooImpl, [number, string]>` where there is a different interface for each operation foo, called `fooImpl` by convention. The reason for the second version is that I have a thought that maybe by avoiding a single giant interface I can avoid some of the complexity of those "declare module" sections or possibly eliminate them altogether. I'll post here when there's new stuff working to look at.
Author
Owner

OK, scratch that bit about a branch with a fooImpl interface for each operation foo. I tried to do things that way, and found that it only made things more complex/convoluted, not less.

On the other hand, I found a way to completely eliminate the declare module boilerplate in each implementation file and replace it with a much shorter, fixed section in each <TYPE>/all.ts module. (So that's only one such file for each type supported by mathjs. Also, the incantation that provides the implementation types for in each all.ts will not need to change as implementations are added to or changed for that type: it picks the types up automatically from the object it was already gathering up from the individual implementation files. So it should work with IDEs.)

I have merged the change into main, because it seems like a clear win. Take a look and let me know if this looks like a reasonable basis for at least continuing with some more types/implementations per our discussion earlier today. I'll wait til I hear back from you here before proceeding, in case you have any more concerns.

(There is actually one slight drawback I can think of: If you want to make a specialized bundle that doesn't have some of the groups of operations, then you couldn't import the all.ts modules, and so the types for the implementations would never get published for the Dependency<'foo', [bigint, bigint]> to pick up. In other words, anyone making a specialized bundle like that would have to incorporate their own analogue of the boilerplate in the all.ts modules somewhere else in their distinct import tree. I am not too worried about this because it is definitely possible to write customized versions of that boilerplate, and making such a specialized bundle is a fairly unusual thing to do so I think it is OK if it becomes a bit more complicated, as long as we clearly document the process.)

OK, scratch that bit about a branch with a `fooImpl` interface for each operation `foo`. I tried to do things that way, and found that it only made things more complex/convoluted, not less. On the other hand, I found a way to completely eliminate the `declare module` boilerplate in each implementation file and replace it with a much shorter, fixed section in each `<TYPE>/all.ts` module. (So that's only one such file for each type supported by mathjs. Also, the incantation that provides the implementation types for in each `all.ts` will not need to change as implementations are added to or changed for that type: it picks the types up automatically from the object it was already gathering up from the individual implementation files. So it should work with IDEs.) I have merged the change into main, because it seems like a clear win. Take a look and let me know if this looks like a reasonable basis for at least continuing with some more types/implementations per our discussion earlier today. I'll wait til I hear back from you here before proceeding, in case you have any more concerns. (There is actually one slight drawback I can think of: If you want to make a specialized bundle that doesn't have some of the groups of operations, then you couldn't import the `all.ts` modules, and so the types for the implementations would never get published for the `Dependency<'foo', [bigint, bigint]>` to pick up. In other words, anyone making a specialized bundle like that would have to incorporate their own analogue of the boilerplate in the `all.ts` modules somewhere else in their distinct import tree. I am not _too_ worried about this because it is definitely possible to write customized versions of that boilerplate, and making such a specialized bundle is a fairly unusual thing to do so I think it is OK if it becomes a bit more complicated, as long as we clearly document the process.)
Author
Owner

Hmm, I can't get Dependency<Name, CallSignature> to work like this. It has real trouble with functions like add(Complex<T>, Complex<T>) that depends on add(T, T). So sadly I think I am giving up on that scheme. Working on another scheme that does involve declaring the types of the implementations alongside them, while still trying to avoid as much redundancy as possible. Hang on, I will let you know when I get another version working.

Hmm, I can't get `Dependency<Name, CallSignature>` to work like this. It has real trouble with functions like `add(Complex<T>, Complex<T>)` that depends on `add(T, T)`. So sadly I think I am giving up on that scheme. Working on another scheme that does involve declaring the types of the implementations alongside them, while still trying to avoid as much redundancy as possible. Hang on, I will let you know when I get another version working.
Author
Owner

OK, please if you can look at branch signature_scheme, which switches to a completely different mechanism for specifying the return types of implementations: You need to write a little type operator that takes you from parameter types to return type, for each implementation. Then you can declare the actual implementation to be of the proper type, and the compiler will check that it is (it did already pick up some potential inconsistencies, such as a subtype of number must include 0 for the zero() operator to be defined on that subtype).

Anyhow, if you can look at that branch signature_scheme and let me know your thoughts. The description of the implementation types is definitely not as simple as the first pass, where you just write the function and let the compiler type it. But I just couldn't get the compiler to extract types from that scheme properly (basically in the end because of the issue posted to stack overflow), whereas I think this new scheme will work for extending to more complicated operations. Note that the usage of an implementation as a dependency has remained exactly the same. This change is entirely about writing more information about the connecton between parameter and return types for implementations at the point where the implementation is provided.

If this looks plausible I will be happy to go ahead and implement more operations, like all of the arithmetic functions in Complex, so we can see better how it goes. Let me know.

OK, please if you can look at branch signature_scheme, which switches to a completely different mechanism for specifying the return types of implementations: You need to write a little type operator that takes you from parameter types to return type, for each implementation. Then you can declare the actual implementation to be of the proper type, and the compiler will check that it is (it did already pick up some potential inconsistencies, such as a subtype of number must include 0 for the zero() operator to be defined on that subtype). Anyhow, if you can look at that branch signature_scheme and let me know your thoughts. The description of the implementation types is definitely not as simple as the first pass, where you just write the function and let the compiler type it. But I just couldn't get the compiler to extract types from that scheme properly (basically in the end because of the issue posted to stack overflow), whereas I think this new scheme will work for extending to more complicated operations. Note that the usage of an implementation as a dependency has remained exactly the same. This change is entirely about writing more information about the connecton between parameter and return types for implementations at the point where the implementation is provided. If this looks plausible I will be happy to go ahead and implement more operations, like all of the arithmetic functions in Complex, so we can see better how it goes. Let me know.
Collaborator

Having to define declare module "../core/Dispatcher" once per module instead of for every function is a big win indeed!

I've had a look at signature_scheme. Defining the implementations and dependency like:

export const add: ImpType<'add', [number, number]> = (a, b) => a + b
export const complex_binary = <T>(t: T, u: T): ImpReturns<'complex', [T,T]> => ({re: t, im: u})
export const zero: ImpType<'zero', [number]> = a => 0

// ... 

export const sqrt = (dep: configDependency & Dependency<'complex', [number, number]>) => { ... }

is very neat and readable! What I'm concerned with is the "magic" that is going on (that the user sort of has to understand), having to rely on a naming convention with underscores, having to define corresponding interfaces like add: Params extends BBinary<number> ? number : never and the need for declare module "../core/Dispatcher" at certain places, and just that the way you write an implementation requires typocomath specific stuff as opposed to say export const add = (a: number, b: number) => a + b.

So yes please it would be great if you could implement a handful of arithmetic functions for to or maybe three data types, and a few generic functions, to get a clear picture on how this will shape up.

I will work out a POC refactoring the original pocomath to TypeScript in a straightforward way to see how that would end up.

Having to define `declare module "../core/Dispatcher"` once per module instead of for every function is a big win indeed! I've had a look at `signature_scheme`. Defining the implementations and dependency like: ```ts export const add: ImpType<'add', [number, number]> = (a, b) => a + b export const complex_binary = <T>(t: T, u: T): ImpReturns<'complex', [T,T]> => ({re: t, im: u}) export const zero: ImpType<'zero', [number]> = a => 0 // ... export const sqrt = (dep: configDependency & Dependency<'complex', [number, number]>) => { ... } ``` is very neat and readable! What I'm concerned with is the "magic" that is going on (that the user sort of has to understand), having to rely on a naming convention with underscores, having to define corresponding interfaces like `add: Params extends BBinary<number> ? number : never` and the need for `declare module "../core/Dispatcher"` at certain places, and just that the way you write an implementation requires typocomath specific stuff as opposed to say `export const add = (a: number, b: number) => a + b`. So yes please it would be great if you could implement a handful of arithmetic functions for to or maybe three data types, and a few generic functions, to get a clear picture on how this will shape up. I will work out a POC refactoring the original `pocomath` to TypeScript in a straightforward way to see how that would end up.
Author
Owner
  1. Glad the declarations when you define an implementation are clear.

  2. Also the TypeScript definitions of these types Dependency<Op, Params>, ImpType<Op, Params>, and ImpReturns<Op, Params> are much simpler than they were in the first scheme because now we're not fighting TypeScript's big weakness with function types. So there is less "magic" to understand.

  3. As for the naming convention with underscores: note that within each subdirectory, where there will basically be just one or occasionally a couple of implementations for a given operation, you don't end up using the underscores at all. They just get added automatically in the all.ts for that subdirectory, to avoid name classes in the big bundle. it's even possible that the name clashes could be avoided by putting the pieces in subobjects as opposed to suffixing their names, if you preferred that; I just hadn't figured out the TypeScript type manipulations to do that, the name suffixes were easy because with template literals the TypeScript type system is pretty good with literal string manipulation. in any case, at the individual implementations level you pretty much don't deal with the underscores: see all the things I add to NumbersReturn<Params> and ComplexReturn<Params> are keyed just by exactly the operation names. So that doesn't seem too bad.

  4. Yes having to write the signature of foo as foo: Params extends [string, number] ? boolean : never instead of foo: (string, number) => boolean is really unfortunate and takes some getting used to. it is by far the biggest drawback to this scheme. it occurs to me now that we could define some helper templates to make this easier/more readable, like foo: HasSignature<Params, [string, number], boolean> I will try that in the next go round.

  5. Yes the greater use of declare module in this version is unfortunate. It looks arcane. I haven't been able to come up with a way to avoid it so far, since it seems to me you need to throw all of the signature descriptions into a single interface, but they are spread across multiple files, and that seems to be how TypeScript handles that. Hopefully sooner or later someone will figure a way to avoid that bit of TypeScript esoterica.

  6. I completely agree it would be nicer to be able to write just add: (a: number, b:number) => a+b. well you can, it's just that you also have to declare the signature another way as well because i couldn't get TypeScript to extract the right type information from the function definition. (it would be fine actually if there were only concrete implementations,its the generic ones like add: <T>(w: Complex<T>, z: Complex<T>)... that cause problems.) And so once you have also described the signature extrinsicaly from the definition, it's important that you put that type explicitly on the definition so that the compiler can check that they are consistent. otherwise the declaration and definition could get out of whack, which would be a real problem.

  7. I agree the name BBinary isn't great. I first thought of HomogeneousBinary, but that's just so darn long. I will try to improve this in the next pass.

  8. Ok, I will add another round of operations/implementations and let you know.

1. Glad the declarations when you define an implementation are clear. 2. Also the TypeScript definitions of these types `Dependency<Op, Params>`, `ImpType<Op, Params>`, and `ImpReturns<Op, Params>` are much simpler than they were in the first scheme because now we're not fighting TypeScript's big weakness with function types. So there is less "magic" to understand. 3. As for the naming convention with underscores: note that within each subdirectory, where there will basically be just one or occasionally a couple of implementations for a given operation, you don't end up using the underscores at all. They just get added automatically in the all.ts for that subdirectory, to avoid name classes in the big bundle. it's even possible that the name clashes could be avoided by putting the pieces in subobjects as opposed to suffixing their names, if you preferred that; I just hadn't figured out the TypeScript type manipulations to do that, the name suffixes were easy because with template literals the TypeScript type system is pretty good with literal string manipulation. in any case, at the individual implementations level you pretty much don't deal with the underscores: see all the things I add to `NumbersReturn<Params>` and `ComplexReturn<Params>` are keyed just by exactly the operation names. So that doesn't seem too bad. 4. Yes having to write the signature of foo as `foo: Params extends [string, number] ? boolean : never` instead of `foo: (string, number) => boolean` is really unfortunate and takes some getting used to. it is by far the biggest drawback to this scheme. it occurs to me now that we could define some helper templates to make this easier/more readable, like `foo: HasSignature<Params, [string, number], boolean>` I will try that in the next go round. 5. Yes the greater use of `declare module` in this version is unfortunate. It looks arcane. I haven't been able to come up with a way to avoid it so far, since it seems to me you need to throw all of the signature descriptions into a single interface, but they are spread across multiple files, and that seems to be how TypeScript handles that. Hopefully sooner or later someone will figure a way to avoid that bit of TypeScript esoterica. 6. I completely agree it would be nicer to be able to write just `add: (a: number, b:number) => a+b`. well you can, it's just that you also have to declare the signature another way as well because i couldn't get TypeScript to extract the right type information from the function definition. (it would be fine actually if there were only concrete implementations,its the generic ones like `add: <T>(w: Complex<T>, z: Complex<T>)...` that cause problems.) And so once you have also described the signature extrinsicaly from the definition, it's important that you put that type explicitly on the definition so that the compiler can check that they are consistent. otherwise the declaration and definition could get out of whack, which would be a real problem. 7. I agree the name BBinary isn't great. I first thought of HomogeneousBinary, but that's just so darn long. I will try to improve this in the next pass. 8. Ok, I will add another round of operations/implementations and let you know.
Collaborator

👍

I've started a POC where I convert pocomath JS straight into "basic" TS. It works like a charm as far as I can see, but I may be overlooking something. Can you have a look at it? Please read the TypeSriptExperiment.md in the experiment/ts branch for explanation:

https://github.com/josdejong/pocomath/blob/experiment/ts/TypeScriptExperiment.md

The repo is a copy of https://code.studioinfinity.org/glen/pocomath with a new branch experiment/ts. It would be handy to push this branch to the original pocomath repo instead if we want to work on it further (right now I can't work on your pocomath and you can't work on mine).

👍 I've started a POC where I convert pocomath JS straight into "basic" TS. It works like a charm as far as I can see, but I may be overlooking something. Can you have a look at it? Please read the `TypeSriptExperiment.md` in the `experiment/ts` branch for explanation: https://github.com/josdejong/pocomath/blob/experiment/ts/TypeScriptExperiment.md The repo is a copy of https://code.studioinfinity.org/glen/pocomath with a new branch `experiment/ts`. It would be handy to push this branch to the original `pocomath` repo instead if we want to work on it further (right now I can't work on your `pocomath` and you can't work on mine).
Author
Owner

Nice efforts! It's really great to have the challenge of an alternative to expose/communicate to each other the strengths and weaknesses (if any) of each approach. I put some comments on that POC in an issue in that repository, and hopefully once I get another batch of functions here we can coalesce on trying to type the same set of implementations so that differences in implementation decisions don't cloud the typing decisions. Using the set here is better because we decided we'd like to stay as close as possible to mathjs implementations in this conversion, and so that's what I am trying to do here (as opposed to pocomath where I didn't worry about it at all).

Nice efforts! It's really great to have the challenge of an alternative to expose/communicate to each other the strengths and weaknesses (if any) of each approach. I put some comments on that POC in an issue in that repository, and hopefully once I get another batch of functions here we can coalesce on trying to type the same set of implementations so that differences in implementation decisions don't cloud the typing decisions. Using the set here is better because we decided we'd like to stay as close as possible to mathjs implementations in this conversion, and so that's what I am trying to do here (as opposed to pocomath where I didn't worry about it at all).
Author
Owner

OK, I have implemented a bunch more implementations (all of complex arithmetic up to sqrt plus anything needed for that). They are committed to branch signature_scheme (haven't merged into main yet in case we want to revisit the first scheme, which definitely looked nicer, to see if it can be made to work with greater TypeScript skill...)

Note that if you clone this repo and do pnpm install (I use the pnpm package manager) then it should work to do npx tsc to try to compile the code. Right now it produces no warnings or errors. Then if you do node obj it will run the compiled code, and you should see it pretending to install all the types and implementations.

Please do feel free to go ahead and try to type those according to the scheme you have in mind and see if you can get it to all compile. I will hang out until you're ready to discuss further by this issue discussion or by video which way the project will go as far as typing scheme.

I will add you as a maintainer on this repo now. Looking forward to settling on a good scheme.

OK, I have implemented a bunch more implementations (all of complex arithmetic up to sqrt plus anything needed for that). They are committed to branch signature_scheme (haven't merged into main yet in case we want to revisit the first scheme, which definitely looked nicer, to see if it can be made to work with greater TypeScript skill...) Note that if you clone this repo and do `pnpm install` (I use the pnpm package manager) then it should work to do `npx tsc` to try to compile the code. Right now it produces no warnings or errors. Then if you do `node obj` it will run the compiled code, and you should see it pretending to install all the types and implementations. Please do feel free to go ahead and try to type those according to the scheme you have in mind and see if you can get it to all compile. I will hang out until you're ready to discuss further by this issue discussion or by video which way the project will go as far as typing scheme. I will add you as a maintainer on this repo now. Looking forward to settling on a good scheme.
Collaborator

👍 yeah let's iterate a bit more on the two approaches, and when all is clear, summarize the main differences in a comparison table and discuss the pros/cons of each.

Let me try to add a section generic/arithmetic.ts in typocomath and implement a generic function square there that dependes on a generic multiply.

👍 yeah let's iterate a bit more on the two approaches, and when all is clear, summarize the main differences in a comparison table and discuss the pros/cons of each. Let me try to add a section `generic/arithmetic.ts` in `typocomath` and implement a generic function `square` there that dependes on a generic multiply.
Collaborator

Ok in the branch signature_scheme_generic I implemented a new function unequal in /numbers/relational.ts, and I tried to implement a generic function square in /generic/arithmetics.ts and I think I'm close but can't figure it out (and I have absolutly no clue on what I'm doing). What am I missing there?

I noticed that the IDE support is not really helpful:

equal_signature

the parameters have a generic name args_0 and args_1 instead of for example x and y, and the return type says ImpReturns<'equal', [number, number]>, which gives me no clue on what the function returns, I would love to see number there.

Ok in the branch [signature_scheme_generic](https://code.studioinfinity.org/glen/typocomath/src/branch/signature_scheme_generic) I implemented a new function `unequal` in `/numbers/relational.ts`, and I tried to implement a generic function `square` in `/generic/arithmetics.ts` and I think I'm close but can't figure it out (and I have absolutly no clue on what I'm doing). What am I missing there? I noticed that the IDE support is not really helpful: ![equal_signature](/attachments/6c059d0b-f015-4ae1-9d59-bfb06be4aa91) the parameters have a generic name `args_0` and `args_1` instead of for example `x` and `y`, and the return type says `ImpReturns<'equal', [number, number]>`, which gives me no clue on what the function returns, I would love to see `number` there.
Author
Owner

yes for square it is the first case of trying to supply a typing that should work for all types T. with no bound at all. it should be possible but as it is something new not anywhere else in typocomath yet it is no surprise it is being a little tricky to work out. I will see if I can get it to type properly. Should I branch again from signature_scheme_generic or add a commit to it? also unequal basically looks good to me so I will see if/why the compiler doesn't like it...

yes for square it is the first case of trying to supply a typing that should work for all types T. with no bound at all. it should be possible but as it is something new not anywhere else in typocomath yet it is no surprise it is being a little tricky to work out. I will see if I can get it to type properly. Should I branch again from signature_scheme_generic or add a commit to it? also unequal basically looks good to me so I will see if/why the compiler doesn't like it...
Collaborator

I will see if I can get it to type properly. Should I branch again from signature_scheme_generic or add a commit to it?

Yes please, feel free to edit the branch and merge it into signature_scheme when ready, then we can throw away signature_scheme_generic.

> I will see if I can get it to type properly. Should I branch again from signature_scheme_generic or add a commit to it? Yes please, feel free to edit the branch and merge it into `signature_scheme` when ready, then we can throw away `signature_scheme_generic`.
Author
Owner

As for ide support, i think you mean instead of ImpReturns<'equal', [number, number]> you would like to see boolean. I agree, I think the compiler has enough information to know that is boolean; you could try defining a variable of that type and assigning 'fred' to it and see what error message you get. as to why the IDE "stops" at ImpReturns, I don't know... maybe it's just limiting the amount of work it does? not really sure how IDEs determine types. I mean, I think they use the compiler under the hood, but how that all interacts is unknown territory for me...

As for ide support, i think you mean instead of `ImpReturns<'equal', [number, number]>` you would like to see `boolean`. I agree, I think the compiler has enough information to know that is boolean; you could try defining a variable of that type and assigning `'fred'` to it and see what error message you get. as to why the IDE "stops" at ImpReturns, I don't know... maybe it's just limiting the amount of work it does? not really sure how IDEs determine types. I mean, I think they use the compiler under the hood, but how that all interacts is unknown territory for me...
Author
Owner

OK, I specified the type of square, verified that all compiled and ran (unequal was in fact totally fine), and merged the branch into signature_scheme and deleted it. Looking forward to your thoughts.

OK, I specified the type of `square`, verified that all compiled and ran (`unequal` was in fact totally fine), and merged the branch into signature_scheme and deleted it. Looking forward to your thoughts.
Author
Owner

Closing in favor of #6 in which we settled on a candidate resolution of these concerns.

Closing in favor of #6 in which we settled on a candidate resolution of these concerns.
glen closed this issue 2022-12-29 15:54:31 +00:00
Sign in to join this conversation.
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: glen/typocomath#1
No description provided.