Status of supplying implementations and dependencies #6

Closed
opened 2022-12-24 16:30:07 +00:00 by glen · 24 comments
Owner

Just want to summarize where things stand with the options for specifying implementations and dependencies in the NewMathJS (pun intended, but it may be a US thing; there was a big primary education program called "New Math" around the time I was a kid). Another motivation is that I have a new possible way forward to offer. I want to get this issue open, so the real summary will be the first comment below.

Just want to summarize where things stand with the options for specifying implementations and dependencies in the NewMathJS (pun intended, but it may be a US thing; there was a big primary education program called "New Math" around the time I was a kid). Another motivation is that I have a new possible way forward to offer. I want to get this issue open, so the real summary will be the first comment below.
Author
Owner

OK, so far we have considered four schemes, with my rough-and-ready assessments of where they stand:

  1. pocomath (JS) Seems to do everything we need. But Jos is concerned not switching to TypeScript in a major rewrite will discourage contributors given the current dominance of TypeScript, whereas Glen is disaffected with a number of design flaws in TypeScript and therefore thinks it will be a fad especially with the rising ease of using many different languages in browser, so Glen would be fine with it.

  2. Too-clever templates. Glen is enamored with this one because he struggled so hard to find a scheme that preserved the typing spirit of current mathjs but successfully and tightly constrained the types of real implementations, but Jos thinks it's just too clever by at least half and arcane and has more or less vetoed it.

  3. WETness (Write Everything Twice, in fact rewrite all signatures at every use): This indubitably works, and doesn't bug Jos too badly but it grates on Glen like fingernails on a chalkboard.

  4. Global interfaces. Represents a shift in typing philosophy for mathjs, which along with the fact that the first pass at this had some permissive/questionable typings, made Glen very dubious at first, and then that dubiousness infected Jos.

Then both of us brainstormed ideas for an option 5 (such brainstorming absolutely could continue) but nothing so far has seemed like a vein of gold to mine.

So I thought it was at least worth exploring how things would turn out if we just embraced the philosophy shift implicit in scheme 4, and tried to pursue it to a full yet constrained typing of everything we have so far.

In that spirit, I offer you approach

4.5) Refined global interfaces

in a new branch for your pulling pleasure, appropriately named approach4.5 and based off of branch approach4_typealias.

To describe it in a couple of nutshells:

  • there is one interface AssociatedTypes<T> that each type added to the system publishes to in order to associate some key related types: right now the type of its zero element, its one element, its not-a-number-element, and its associated real-number type. (I imagine in the future there might be an associated "element type" for collection types, etc.)
  • there is another interface Operations<T> to which one property is published for each operation, giving the operation's global signature pattern for "operating on type T". In other words, its generic function type, just like the type aliases from the first draft approach4, but this time with the parameters and return type specified separately to make it easy for TypeScript to manipulate them.
  • A few simple type macros that provide easy, concise, non-redundant access to all this information. Note that putting the signatures in the Operations interface means that TypeScript knows the name of each operation. That means we don't need separate type aliases for each operation, as they can be used by name, avoiding having the big import list of lots of type aliases in the arithmetic files that was redundant with the list of operations implemented.

To my pleasure, it did not feel too badly constraining to have these global signatures for the operations, and having the information laid out in the two interfaces felt very clear and systematic to me. I was also able to get nice tight typings for all of the current batch of implementations and their dependencies. The real proof that this typing approach was pulling its weight was that the process detected a subtle typing error in the complex implementation of sqrt that none of the other approaches did: the prior code was trying to automatically convert from Complex<RealType<T>> to T, but because of possible nesting those might not be the same type! Fixing this actually simplified the implementation and allowed the elimination of an entire auxiliary function. So this experience has actually made me feel quite encouraged about refining and pursuing approach 4.

There is one kink worth mentioning: as I suspected and raised as a possible concern, there definitely are some operations that have more than one necessary pattern of parameter and return types, even allowing for a template type T in the signature. The specific example that has come up so far is add, where you need to be able to add two entities of the same type to get an entity of that type, and you need to be able to add a real number to a complex number to get a complex number. The latter is definitely a distinct signature that needs to be referred to, and just as definitely should end up as nothing more another implementation for add, since it can be completely distinguished by the input types and is doing the mathematically same operation.

So for now to get something working, I used the tactic of creating the other pattern as if it was another operation, but giving it a name that consists of add with a suffix that starts with _, namely "add_real" (instead of the former addReal, before I realized it was just a case of add). And then I am presuming that the Dispatcher will strip _... suffixes, coalescing them into a single mathjs operation. I am completely open to another mechanism that you might think of, or I am actually fine to go with this coalescing at the mathjs bundle level by matching up names, I don't think it's too terrible or too confusing. But I freely admit it's a bit clunky and might have pitfalls as a naming convention; on the other hand, it seems fairly safe as underscore is currently not used in any mathjs operation names.

So Merry Christmas and looking forward to your thoughts when you are ready to get back to this; I am cautiously optimistic. Let me know when you've had a chance to absorb and we will set a video chat time.

OK, so far we have considered four schemes, with my rough-and-ready assessments of where they stand: 1) pocomath (JS) Seems to do everything we need. But Jos is concerned not switching to TypeScript in a major rewrite will discourage contributors given the current dominance of TypeScript, whereas Glen is disaffected with a number of design flaws in TypeScript and therefore thinks it will be a fad especially with the rising ease of using many different languages in browser, so Glen would be fine with it. 2) Too-clever templates. Glen is enamored with this one because he struggled so hard to find a scheme that preserved the typing spirit of current mathjs but successfully and tightly constrained the types of real implementations, but Jos thinks it's just too clever by at least half and arcane and has more or less vetoed it. 3) WETness (Write Everything Twice, in fact rewrite all signatures at every use): This indubitably works, and doesn't bug Jos too badly but it grates on Glen like fingernails on a chalkboard. 4) Global interfaces. Represents a shift in typing philosophy for mathjs, which along with the fact that the first pass at this had some permissive/questionable typings, made Glen very dubious at first, and then that dubiousness infected Jos. Then both of us brainstormed ideas for an option 5 (such brainstorming absolutely could continue) but nothing so far has seemed like a vein of gold to mine. So I thought it was at least worth exploring how things would turn out if we just embraced the philosophy shift implicit in scheme 4, and tried to pursue it to a full yet constrained typing of everything we have so far. In that spirit, I offer you approach 4.5) Refined global interfaces in a new branch for your pulling pleasure, appropriately named `approach4.5` and based off of branch `approach4_typealias`. To describe it in a couple of nutshells: - there is one interface `AssociatedTypes<T>` that each type added to the system publishes to in order to associate some key related types: right now the type of its zero element, its one element, its not-a-number-element, and its associated real-number type. (I imagine in the future there might be an associated "element type" for collection types, etc.) - there is another interface `Operations<T>` to which one property is published for each operation, giving the operation's global signature pattern for "operating on type T". In other words, its generic function type, just like the type aliases from the first draft approach4, but this time with the parameters and return type specified separately to make it easy for TypeScript to manipulate them. - A few simple type macros that provide easy, concise, non-redundant access to all this information. Note that putting the signatures in the Operations interface means that TypeScript knows the name of each operation. That means we don't need separate type aliases for each operation, as they can be used by name, avoiding having the big import list of lots of type aliases in the arithmetic files that was redundant with the list of operations implemented. To my pleasure, it did not feel too badly constraining to have these global signatures for the operations, and having the information laid out in the two interfaces felt very clear and systematic to me. I was also able to get nice tight typings for all of the current batch of implementations and their dependencies. The real proof that this typing approach was pulling its weight was that the process detected a subtle typing error in the complex implementation of sqrt that none of the other approaches did: the prior code was trying to automatically convert from `Complex<RealType<T>>` to `T`, but because of possible nesting those might not be the same type! Fixing this actually simplified the implementation and allowed the elimination of an entire auxiliary function. So this experience has actually made me feel quite encouraged about refining and pursuing approach 4. There is one kink worth mentioning: as I suspected and raised as a possible concern, there definitely are some operations that have more than one necessary pattern of parameter and return types, even allowing for a template type T in the signature. The specific example that has come up so far is `add`, where you need to be able to add two entities of the same type to get an entity of that type, and you need to be able to add a real number to a complex number to get a complex number. The latter is definitely a distinct signature that needs to be referred to, and just as definitely should end up as nothing more another implementation for add, since it can be completely distinguished by the input types and is doing the mathematically same operation. So for now to get something working, I used the tactic of creating the other pattern as if it was another operation, but giving it a name that consists of `add` with a suffix that starts with `_`, namely "add_real" (instead of the former addReal, before I realized it was just a case of `add`). And then I am presuming that the Dispatcher will strip `_...` suffixes, coalescing them into a single mathjs operation. I am completely open to another mechanism that you might think of, or I am actually fine to go with this coalescing at the mathjs bundle level by matching up names, I don't think it's too terrible or too confusing. But I freely admit it's a bit clunky and might have pitfalls as a naming convention; on the other hand, it seems _fairly_ safe as underscore is currently not used in any mathjs operation names. So Merry Christmas and looking forward to your thoughts when you are ready to get back to this; I am cautiously optimistic. Let me know when you've had a chance to absorb and we will set a video chat time.
Author
Owner

My apologies for increasing the collection of options on the table, but I have just pushed for your consideration the branch approach2.5. Basically, using the patterns and tricks developed for approach4.5 (where a global interface pattern is specified for each operation in an interfaces subdirectory) it is possible to specify the type of each implementation alongside it in the same file in ordinary (a: SomeType, b: OtherType) => RetType notation, without the assumption that there is a single universal type pattern for that operation. Since this doesn't change the typing philosophy of mathjs (the type of each operation is really just the accumulation of the types of all of its implementations), I am calling this option 2.5. However, unlike its ancestor (approach 2, "Too Clever", in branch signature_scheme) it is quite straightforward to declare the type of an implementation: there is just an interface in which one puts its "ordinary" typescript function type, under the same name as the implementation. Then by declaring each implementation 'foo' to return ImpType<'foo', [Arg1Type, Arg2Type]> the compiler at least checks that the specified type of the implementation and the code of the implementation are compatible.

Two advantages of approach2.5 over approach4.5: (a) as mentioned, the typing philosophy does not change; (b) since we are not trying to define global types for everything, there is no need for the AssociatedTypes<T> structure. And to my eye, these advantages come at not very much cost in terms of the underlying complexity of the implementation. Also, I think the type declarations for the implementations are every bit as readable/writable as in approach 4.5.

My apologies for increasing the collection of options on the table, but I have just pushed for your consideration the branch `approach2.5`. Basically, using the patterns and tricks developed for approach4.5 (where a global interface pattern is specified for each operation in an `interfaces` subdirectory) it is possible to specify the type of each implementation alongside it in the same file in ordinary `(a: SomeType, b: OtherType) => RetType` notation, _without_ the assumption that there is a single universal type pattern for that operation. Since this doesn't change the typing philosophy of mathjs (the type of each operation is really just the accumulation of the types of all of its implementations), I am calling this option 2.5. However, unlike its ancestor (approach 2, "Too Clever", in branch signature_scheme) it is quite straightforward to declare the type of an implementation: there is just an interface in which one puts its "ordinary" typescript function type, under the same name as the implementation. Then by declaring each implementation 'foo' to return `ImpType<'foo', [Arg1Type, Arg2Type]>` the compiler at least checks that the specified type of the implementation and the code of the implementation are compatible. Two advantages of approach2.5 over approach4.5: (a) as mentioned, the typing philosophy does not change; (b) since we are not trying to define global types for everything, there is no need for the `AssociatedTypes<T>` structure. And to my eye, these advantages come at not very much cost in terms of the underlying complexity of the implementation. Also, I think the type declarations for the implementations are every bit as readable/writable as in approach 4.5.
Collaborator

😂 you're getting poetic.

I like the approaches 4.5 and 2.5 a lot, thanks! Feels like we're getting there. Let's discuss today in a video call.

😂 you're getting poetic. I like the approaches 4.5 and 2.5 a lot, thanks! Feels like we're getting there. Let's discuss today in a video call.
Author
Owner

Per our conversation today, we are going to try to move ahead with "approach 4.6" which is essentially like approach 4.5 (including the philosophy of a single overarching consistent interface for each operation, with any "exceptions" that cannot be told apart by the input types being mandatory to be a different operation, i.e. have a differtent name). The differences will be in the syntax and naming choices of supplying interfaces, dependencies, and implementation:

//interfaces/arithmetic.ts:
interface Operations<T> {
    add: (a: T, b: T) => T
    addReal: AliasOf<'add'>(a: T, b: RealType<T>) => T // OK, no conflict with T,T=>T
    multiply: (a: T, b: T) => T
    multiplyOne: AliasOf<'multiply', (arg: T) => T >
    multiplySeveral: AliasOf<'multiply',
        (arg1: T, arg2: T, arg3: T, ...rest: T[]) => T>
    // or if we get ambitious about making multiplySeveral more like current mathjs,
    // maybe the Ts all become `any`; or if even more ambitions, maybe they
    // are all different types and there is a type operator that figures it out, but
    // I am not too hopeful we can pull that off...
    multiplyMatrixVec: AliasOf<'multiply', (a: Matrix, b: Vec) => Vec> // OK...
    // ... because Matrix and Vec are different types, so doesn't conflict with
    // ... the T,T signature
    dotProduct: (a: Vec: b: Vec) => number // has to be different name, not ...
    // ... an alias, because of conflict with T,T => T
    square: (a: T) => Returns<'multiply', T>
}

//Complex/arithmetic.ts: (not the actual dependencies, just for illustration)
export const multiply = 
   <T>(dep: Dependencies<'multiply', T> 
            & Dependencies<'addReal', T>): Signature<'multiply', Complex<T>> =>
   (w, z) => ... implementation details ..

Note the switch to using direct function types instead of {params: [T1, T2], returns: T3}, a type operator for aliases, and the names Signature and Returns instead of OpType and OpReturns.

One slight remaining question that we did not address: when using an aliased operation like addReal, do you refer just to the alias (as I did in the above example), just to the operation name it was aliased to (like we would say 'add' instead of 'addReal' to express the dependencies in the implementation of another operation, so you wouldn't have to worry about whether you were getting the original or an alias) or somehow both/a slightly different dependency operator that flagged that you were looking for an alias?

Finally, one possible implementation of the AliasOf type operation, but this is an implementation detail and it could be any other type so long as we can extract the alias from it:

type AliasOf<Name extends string, T> = T & {aliasOf?: Name}

I think that's everything we settled on but if I have missed something feel free to add another comment here. Also, if either of us starts work on revising the current approach4.5 branch to this 4.6 syntax, or on the next step of trying rtti, that person should leave a note here and we should both look for such notes to avoid duplicating effort. Thanks!

Per our conversation today, we are going to try to move ahead with "approach 4.6" which is essentially like approach 4.5 (including the philosophy of a single overarching consistent interface for each operation, with any "exceptions" that cannot be told apart by the input types being mandatory to be a different operation, i.e. have a differtent name). The differences will be in the syntax and naming choices of supplying interfaces, dependencies, and implementation: ``` //interfaces/arithmetic.ts: interface Operations<T> { add: (a: T, b: T) => T addReal: AliasOf<'add'>(a: T, b: RealType<T>) => T // OK, no conflict with T,T=>T multiply: (a: T, b: T) => T multiplyOne: AliasOf<'multiply', (arg: T) => T > multiplySeveral: AliasOf<'multiply', (arg1: T, arg2: T, arg3: T, ...rest: T[]) => T> // or if we get ambitious about making multiplySeveral more like current mathjs, // maybe the Ts all become `any`; or if even more ambitions, maybe they // are all different types and there is a type operator that figures it out, but // I am not too hopeful we can pull that off... multiplyMatrixVec: AliasOf<'multiply', (a: Matrix, b: Vec) => Vec> // OK... // ... because Matrix and Vec are different types, so doesn't conflict with // ... the T,T signature dotProduct: (a: Vec: b: Vec) => number // has to be different name, not ... // ... an alias, because of conflict with T,T => T square: (a: T) => Returns<'multiply', T> } //Complex/arithmetic.ts: (not the actual dependencies, just for illustration) export const multiply = <T>(dep: Dependencies<'multiply', T> & Dependencies<'addReal', T>): Signature<'multiply', Complex<T>> => (w, z) => ... implementation details .. ``` Note the switch to using direct function types instead of `{params: [T1, T2], returns: T3}`, a type operator for aliases, and the names Signature and Returns instead of OpType and OpReturns. One slight remaining question that we did not address: when **using** an aliased operation like addReal, do you refer just to the alias (as I did in the above example), just to the operation name it was aliased **to** (like we would say 'add' instead of 'addReal' to express the dependencies in the implementation of another operation, so you wouldn't have to worry about whether you were getting the original or an alias) or somehow both/a slightly different dependency operator that flagged that you were looking for an alias? Finally, one possible implementation of the AliasOf type operation, but this is an implementation detail and it could be any other type so long as we can extract the alias from it: ``` type AliasOf<Name extends string, T> = T & {aliasOf?: Name} ``` I think that's everything we settled on but if I have missed something feel free to add another comment here. Also, if either of us starts work on revising the current approach4.5 branch to this 4.6 syntax, or on the next step of trying rtti, that person should leave a note here and we should both look for such notes to avoid duplicating effort. Thanks!
Author
Owner

Oh one more thing from our notes: ultimately we will also export a pure javascript method for registering an additional implementation, where the type will have to be specified by some JavaScript structure: a string that is parsed, or an object with particular keys, or something. I think the idea is that in the main TypeScript internals, it just combs through all the exported functions and uses typescript-rtti to infer all that information and generates the call to that same JavaScript registration function.

Oh one more thing from our notes: ultimately we will also export a pure javascript method for registering an additional implementation, where the type will have to be specified by some JavaScript structure: a string that is parsed, or an object with particular keys, or something. I think the idea is that in the main TypeScript internals, it just combs through all the exported functions and uses typescript-rtti to infer all that information and generates the call to that same JavaScript registration function.
Author
Owner

One slight remaining question that we did not address: when using an aliased operation like addReal, do you refer just to the alias (as I did in the above example), just to the operation name it was aliased to (like we would say 'add' instead of 'addReal' to express the dependencies in the implementation of another operation, so you wouldn't have to worry about whether you were getting the original or an alias) or somehow both/a slightly different dependency operator that flagged that you were looking for an alias?

@josdejong I think it would be good to reach a tentative mutually agreeable proposal on this remaining small point before either of us tries to actually get approach4.6 running. I don't immediately have a strong sense of what would be ideal here, so if you have any thoughts please share... thanks and thanks again for a good conversation today.

> One slight remaining question that we did not address: when **using** an aliased operation like addReal, do you refer just to the alias (as I did in the above example), just to the operation name it was aliased to (like we would say 'add' instead of 'addReal' to express the dependencies in the implementation of another operation, so you wouldn't have to worry about whether you were getting the original or an alias) or somehow both/a slightly different dependency operator that flagged that you were looking for an alias? @josdejong I think it would be good to reach a tentative mutually agreeable proposal on this remaining small point before either of us tries to actually get approach4.6 running. I don't immediately have a strong sense of what would be ideal here, so if you have any thoughts please share... thanks and thanks again for a good conversation today.
Author
Owner

i thought about it some more and realized two things:

  1. in the "standard form" Dependencies<'addReal', Complex<number>> you couldn't use only the 'add' operation name -- since we're only supplying one type, not the whole signature, the Dispatcher would have no way of knowing that you wanted the one that adds a real. that is after all the point of having the aliases, to provide access to non-conflicting alternate signatures. And if you do mention addReal, there's no need to specify in the dependency that it's an alias of add, that's already been specified in the interface. so I think the dependency should look just the way I wrote it in the summary above.

  2. We also discussed that the dispatcher should allow a dependency as a full written out function type with all parameters and return type, would request input and output conversions if necessary. So the 4.6 mockup should have an instance of this, and for that, the key in the dependencies should be the "real" operation name (e.g. 'add') so the dispatcher knows which operation you're requesting this specialized instance for.

if you agree with these items then either of us can go ahead with revasion 4.6 when we happen to have chance.

i thought about it some more and realized two things: 1) in the "standard form" `Dependencies<'addReal', Complex<number>>` you couldn't use only the 'add' operation name -- since we're only supplying one type, not the whole signature, the Dispatcher would have no way of knowing that you wanted the one that adds a real. that is after all the point of having the aliases, to provide access to non-conflicting alternate signatures. And if you do mention addReal, there's no need to specify in the dependency that it's an alias of add, that's already been specified in the interface. so I think the dependency should look just the way I wrote it in the summary above. 2) We also discussed that the dispatcher should allow a dependency as a full written out function type with all parameters and return type, would request input and output conversions if necessary. So the 4.6 mockup should have an instance of this, and for that, the key in the dependencies should be the "real" operation name (e.g. 'add') so the dispatcher knows which operation you're requesting this specialized instance for. if you agree with these items then either of us can go ahead with revasion 4.6 when we happen to have chance.
Collaborator

Good summary. Agree, lets work this out in revision 4.6.

afbeelding

About aliases: thinking aloud here. In some cases, like say add and addReal, it can be that you need both these dependencies. I that case you need to use unique names like add and addReal. However, if you only need the alias as a dependency, for example multiplySeveral, it would be awkward if you need to know (and use) the correct alias name instead of simply multiply (and I think the latter is the common case, the former the edge case). So... maybe we should see if it is possible to allow using both names, depending on your needs? Or, always use the original name like multiply, and introduce an additional alias solution at the Dependencies side so you have freedom to rename functions? I'm afraid though this will give difficulties in TS 🤔

Good summary. Agree, lets work this out in revision 4.6. ![afbeelding](/attachments/6e4bc774-a7de-484d-80d5-a5a380c85ac6) About aliases: thinking aloud here. In some cases, like say `add` and `addReal`, it can be that you need _both_ these dependencies. I that case you need to use unique names like `add` and `addReal`. However, if you _only_ need the alias as a dependency, for example `multiplySeveral`, it would be awkward if you need to know (and use) the correct alias name instead of simply `multiply` (and I think the latter is the common case, the former the edge case). So... maybe we should see if it is possible to allow using _both_ names, depending on your needs? Or, always use the original name like `multiply`, and introduce an additional alias solution at the Dependencies side so you have freedom to rename functions? I'm afraid though this will give difficulties in TS 🤔
Author
Owner

ah, thanks for your excellent observation, as it uncovers a bigger problem. what is the notation when you are using an operation that does not have any aliased implementations but you need it for two different types, both as dependencies of the same function? this has already come up. with the "naming convention" solution in 4.5, you just made the key start with the operation name. without that convention, there does need to be some aliasing mechanism at the dependency site so that you can have two different keys in the dependencies that refer to different implementations of the same operation. So I am afraid we need a proposal here before we can proceed.

ah, thanks for your excellent observation, as it uncovers a bigger problem. what is the notation when you are using an operation that does not have any aliased implementations but you need it for two different types, both as dependencies of the same function? this has already come up. with the "naming convention" solution in 4.5, you just made the key start with the operation name. without that convention, there does need to be some aliasing mechanism at the dependency site so that you can have two different keys in the dependencies that refer to different implementations of the same operation. So I am afraid we need a proposal here before we can proceed.
Author
Owner

OK, here's a proposal. I can think of a bunch of possible use cases, some of which have already occurred, and some of which are hypothetical.

Dependencies<'add', number>

The standard case: you get the main signature for add, and the key in the dep object is add, and of course the dispatcher knows you want add.

Dependencies<'addReal', Complex<number>>

Using an alternate signature for an operation under its alternate name: the key in the dep object is addReal, and the dispatcher knows you want add because addReal is already an alias for add.

Dependencies<'add', Complex<number>>
& {addNumbers: Signature<'add', number>}

You need add both on complex numbers and on number itself. So you have to give one a distinct name. It knows you mean the add operation in the second version because of the 'add' argument to Signature. This sort of combination might be discouraged (although it should work); it might be clearer to write

{addComplex: Signature<'add', Complex<number>>, 
 addNumbers: Signature<'add', number>}

so that all of the uses in the function body will be clearly marked as dep.addComplex or dep.addNumbers so that reader of the code doesn't have to keep track of "which signature did I call just 'add' by itself?"

{add: Signature<'addReal', Complex<number>}

You only need the addReal signature of add, but you'd like to use it as dep.add. Signature gets the correct type, and the dispatcher knows it's add because of the key and/or because addReal is already an alias for add. Again, not 100% sure this should be encouraged, as the uses of dep.add in the function body might then be a bit misleading. Or maybe it would be clear enough, if this is the only version of add used in a particular function.

{add: (r:number, z:Complex<number>) => Complex<BigNumber>}

You want an "adapted" implementation of add that uses conversions in input and output, and you are using it under the name 'add'. As the type for this key is a plain function type, the dispatcher knows you want add because of the key. If this comes up, using the plain key add should probably be considered OK practice, since if the key is not add either, the operation you want would have to be somewhat verbosely specified, as shown in the next example.

Dependencies<'add', number> &
{addAndMakeComplex: Implementation<'add', (a: number, b: number) => Complex<number>>}

You want the standard signature of add under its standard name, but you also need a special one with adapters under a special name. Then you need to wrap the plain function type in a type operator that will tell the dispatcher you want an implementation of 'add'; otherwise it will have no way of knowing which operation you're looking for. The name Implementation<Name, FuncType> is just a suggestion for the name of this type operator, open to other names if you think something else would be better.

That covers all of the possibilities I could think of at the moment.

Finally, in the pattern {baz: Implementation<'foo', Signature<'bar', bigint>>}, if it is ever needed, the name 'foo' would take precedence over both 'bar' and 'baz' in terms of what operation the dispatcher obtains an implementation for as the dependency. That way if any of the above cases turns out to be ambiguous in practice and/or there are other edge cases that I haven't thought of, there will be a mechanism that always allows you to control exactly what operation you are depending on.

This all may seem a bit complicated but in practice I think the first two cases, which are quite straightforward, will cover >95% of the actual dependency specifications. We just need mechanisms for the unusual instances where those standard patterns aren't enough. Let me know how that all sounds and if I have missed any cases that you think need attention, or if the proposed syntax doesn't feel comfortable in any of the cases I've covered. Thanks.

OK, here's a proposal. I can think of a bunch of possible use cases, some of which have already occurred, and some of which are hypothetical. * ``` Dependencies<'add', number> ``` The standard case: you get the main signature for add, and the key in the dep object is add, and of course the dispatcher knows you want add. * ``` Dependencies<'addReal', Complex<number>> ``` Using an alternate signature for an operation under its alternate name: the key in the dep object is addReal, and the dispatcher knows you want add because addReal is already an alias for add. * ``` Dependencies<'add', Complex<number>> & {addNumbers: Signature<'add', number>} ``` You need add both on complex numbers and on number itself. So you have to give one a distinct name. It knows you mean the add operation in the second version because of the 'add' argument to Signature. This sort of combination might be discouraged (although it should work); it might be clearer to write ``` {addComplex: Signature<'add', Complex<number>>, addNumbers: Signature<'add', number>} ``` so that all of the uses in the function body will be clearly marked as `dep.addComplex` or `dep.addNumbers` so that reader of the code doesn't have to keep track of "which signature did I call just 'add' by itself?" * ``` {add: Signature<'addReal', Complex<number>} ``` You only need the addReal signature of add, but you'd like to use it as `dep.add`. Signature gets the correct type, and the dispatcher knows it's add because of the key and/or because addReal is already an alias for add. Again, not 100% sure this should be encouraged, as the uses of `dep.add` in the function body might then be a bit misleading. Or maybe it would be clear enough, if this is the only version of add used in a particular function. * ``` {add: (r:number, z:Complex<number>) => Complex<BigNumber>} ``` You want an "adapted" implementation of add that uses conversions in input and output, and you are using it under the name 'add'. As the type for this key is a plain function type, the dispatcher knows you want add because of the key. If this comes up, using the plain key `add` should probably be considered OK practice, since if the key is not `add` either, the operation you want would have to be somewhat verbosely specified, as shown in the next example. * ``` Dependencies<'add', number> & {addAndMakeComplex: Implementation<'add', (a: number, b: number) => Complex<number>>} ``` You want the standard signature of add under its standard name, but you also need a special one with adapters under a special name. Then you need to wrap the plain function type in a type operator that will tell the dispatcher you want an implementation of 'add'; otherwise it will have no way of knowing which operation you're looking for. The name `Implementation<Name, FuncType>` is just a suggestion for the name of this type operator, open to other names if you think something else would be better. That covers all of the possibilities I could think of at the moment. Finally, in the pattern `{baz: Implementation<'foo', Signature<'bar', bigint>>}`, if it is ever needed, the name 'foo' would take precedence over both 'bar' and 'baz' in terms of what operation the dispatcher obtains an implementation for as the dependency. That way if any of the above cases turns out to be ambiguous in practice and/or there are other edge cases that I haven't thought of, there will be a mechanism that always allows you to control exactly what operation you are depending on. This all may seem a bit complicated but in practice I think the first two cases, which are quite straightforward, will cover >95% of the actual dependency specifications. We just need mechanisms for the unusual instances where those standard patterns aren't enough. Let me know how that all sounds and if I have missed any cases that you think need attention, or if the proposed syntax doesn't feel comfortable in any of the cases I've covered. Thanks.
Collaborator

Thanks, this makes sense. Agree

The following notation reads very clear to me:

{addComplex: Signature<'add', Complex<number>>, 
 addNumbers: Signature<'add', number>}

We can maybe extend the Dependencies API to allow for an alias name (just syntax sugar for the example above):

Dependencies<'addComplex', 'add', Complex<number>> 
& Dependencies<'addNumbers', 'add', number>

@glen I'll work out the 4.6 approach today

Thanks, this makes sense. Agree The following notation reads very clear to me: ```ts {addComplex: Signature<'add', Complex<number>>, addNumbers: Signature<'add', number>} ``` We can maybe extend the `Dependencies` API to allow for an alias name (just syntax sugar for the example above): ```ts Dependencies<'addComplex', 'add', Complex<number>> & Dependencies<'addNumbers', 'add', number> ``` @glen I'll work out the 4.6 approach today
Author
Owner

Great, I look forward to the next to iteration. But please, can we keep Dependencies<'add', number> simple and just have it handle the very common case that the operation we want to depend on and the key we will refer to it by are the same? that way we can preserve the nice distributivity when you want more than one dependency with the same type and avoid it getting too complex/confusing. I think the case when the operation and the key need to be different will be very rare and it will be fine - in fact clearer- for a different syntax to be needed then. thanks for considering.

Great, I look forward to the next to iteration. But please, can we keep `Dependencies<'add', number>` simple and just have it handle the very common case that the operation we want to depend on and the key we will refer to it by are the same? that way we can preserve the nice distributivity when you want more than one dependency with the same type and avoid it getting too complex/confusing. I think the case when the operation and the key need to be different will be very rare and it will be fine - in fact clearer- for a different syntax to be needed then. thanks for considering.
Collaborator

I've pushed a branch approach4.6

I've tried an additional API like DependencyAlias<'addComplex', 'add', Complex<number>>, but I do not really like it. It is yet another API, and I prefer keeping things simple, so we have either Dependencies<...> (simple and compact), or you create your own object { name1: Signature<...>, name2: ... } (flexible).

There is one unresolved issue left I think: in complex/arightmetics.ts you see sqrt is defining:

    & {
         addNumber: Signature<'addReal', T>, // TODO: should use Signature<'add'> here
         addReal: Signature<'add', RealType<T>>,
         addComplex: Signature<'addReal', Complex<T>> // TODO: should use Signature<'add'> here
      }

Here, we should use the original name add instead of the alias name addReal. I din't manage to figure that out yet (AliasOf should result in an object with two keys? that results in an object with duplicate keys though and conflicts 🤔).

I've pushed a branch [`approach4.6`](https://code.studioinfinity.org/glen/typocomath/src/branch/approach4.6) I've tried an additional API like `DependencyAlias<'addComplex', 'add', Complex<number>>`, but I do not really like it. It is yet another API, and I prefer keeping things simple, so we have either `Dependencies<...>` (simple and compact), or you create your own object `{ name1: Signature<...>, name2: ... }` (flexible). There is one unresolved issue left I think: in `complex/arightmetics.ts` you see `sqrt` is defining: ```ts & { addNumber: Signature<'addReal', T>, // TODO: should use Signature<'add'> here addReal: Signature<'add', RealType<T>>, addComplex: Signature<'addReal', Complex<T>> // TODO: should use Signature<'add'> here } ``` Here, we should use the original name `add` instead of the alias name `addReal`. I din't manage to figure that out yet (`AliasOf` should result in an object with two keys? that results in an object with duplicate keys though and conflicts 🤔).
Author
Owner

This branch looks fine to me. I am confused as to why you say that addNumber and addComplex should use Signature<'add',...>. I might quibble with the names, but the standard add signature is to take two items of type T and add them to get an item of type T. but that's not what either of these need to do: they need to add an element of a real type to some other type. Referring to that signature of 'add' is precisely why the 'addReal' alias was created, so it seems eminently appropriate to use that alias in these two dependencies. I might suggest changing the names to 'addTReal' and 'addComplexReal', though, to make their purposes clearer. especially I worry addNumber is confusing since if this is used for bigint, there won't be any actual number operands involved here at all.

This branch looks fine to me. I am confused as to why you say that addNumber and addComplex should use `Signature<'add',...>`. I might quibble with the names, but the standard `add` signature is to take two items of type T and add them to get an item of type T. but that's not what either of these need to do: they need to add an element of a real type to some other type. Referring to that signature of 'add' is precisely why the 'addReal' alias was created, so it seems eminently appropriate to use that alias in these two dependencies. I might suggest changing the names to 'addTReal' and 'addComplexReal', though, to make their purposes clearer. especially I worry addNumber is confusing since if this is used for bigint, there won't be any actual number operands involved here at all.
Author
Owner

just to expand slightly, if we did use Signature<'add', T> for the one currently called addNumber, how would the compiler (or the reader) know we meant the one that adds a T and a RealType<T> or the one that adds two Ts? that's what aliases were introduced for. Or we could go back to specifying the full argument tuple in all Dependencies, i.e. Dependencies<'add', [T, RealType<T>]> and Dependencies<'add', [RealType<T>, RealType<T>]> here. I'd be fine with that, but you didn't seem to like it very well. Sometimes depending on aliases is the cost of not always having to list out the full argument tuple.

just to expand slightly, if we did use `Signature<'add', T>` for the one currently called addNumber, how would the compiler (or the reader) know we meant the one that adds a T and a `RealType<T>` or the one that adds two Ts? that's what aliases were introduced for. Or we could go back to specifying the full argument tuple in all Dependencies, i.e. `Dependencies<'add', [T, RealType<T>]>` and `Dependencies<'add', [RealType<T>, RealType<T>]>` here. I'd be fine with that, but you didn't seem to like it very well. Sometimes depending on aliases is the cost of not always having to list out the full argument tuple.
Collaborator

ah, yeah, you're right. So, yes, the alternative is to come up with an API specifying individual parameters again, like Dependencies<'add', [T, T]>.

I'm ok with the syntax of both notations Dependencies<'add', T> and Dependencies<'add', [T, T]>. But what I like is to have a solution that does not require advanced TS, where you get TS errors and IDE warnings directly when entering a signature that doesn't exist (and where you get autocompletion in your IDE). That was not the case in the earlier approach in branch signature_scheme that used the Dependencies<'add', [T, T]> notation.

I have to think about this a bit: if turns out that we need aliases only rarely, it is perfect like it is now. But if it turns out that we need them a lot, I would like think this a bit more though. I want to go briefly through the source code of mathjs to get a better insight in this.

ah, yeah, you're right. So, yes, the alternative is to come up with an API specifying individual parameters again, like `Dependencies<'add', [T, T]>`. I'm ok with the _syntax_ of both notations `Dependencies<'add', T>` and `Dependencies<'add', [T, T]>`. But what I like is to have a solution that does not require advanced TS, where you get TS errors and IDE warnings directly when entering a signature that doesn't exist (and where you get autocompletion in your IDE). That was not the case in the earlier approach in branch `signature_scheme` that used the `Dependencies<'add', [T, T]>` notation. I have to think about this a bit: if turns out that we need aliases only rarely, it is perfect like it is now. But if it turns out that we need them a lot, I would like think this a bit more though. I want to go briefly through the source code of mathjs to get a better insight in this.
Author
Owner

Hmmm, well, whether an individual revision I make plays nicely with a particular IDE is hit or miss, since I am not using any sophisticated IDE; and I imagine it may even be hit or miss for you with respect to some other IDE than the one you are using (I don't know how equivalent they are or are not).

But anyhow if you want to leave Dependencies<'add', T> alone and rather than use aliases in Dependencies have another type operator DependencyWithParams<'add', [T, RealType<T>]> for the cases where one needs an implementation on add with a specific parameter tuple other than the "usual" one, that's fine with me. It should be pretty clear how to implement it in the existing framework of 4.6, or I am happy to take a stab if you like. Given that Dependencies is playing nice with your IDE, I can imagine that an analogous DependencyWithParams (feel to use a better name if you can think of one) implemented along the same lines would work fine, too.

A slightly odd thing if we go this route is that then defining interfaces and implementations will be a bit less parallel with using them as dependencies. I don't see any way around using another identifier like addReal and an AliasOf type operator to include more interfaces for the add operation. So if we have to define aliases to create implementations, why not use them when declaring dependencies to indicate that it's this other interface we want?

So my vote is for leaving 4.6 as it is (likely changing the keys for the additional instances of 'add' in Complex sqrt to make them a little clearer).

That said, there's no real problem other than enlarging the API with having something like DependencyWithParams<> too. I just don't see a way to do something similar on the defining side, and I worry slightly that it might become confusing when to use Dependencies<'foo', Type>, when to use DependencyWithParams<'foo', [TypeA, TypeB]>, and when to use Implementation<'foo', (c: TypeC, d: TypeD) => RetType> (the notation we added in the conversation above to request input and output conversions -- we have not actually used in practice yet, but if we want to support it, we should find a real use case in which it would truly be the best way to do things, and include that in the prototype as soon as possible. I am fine with it in principle as we planned but have not yet thought of an actual instance in which it would be used...).

Hmmm, well, whether an individual revision I make plays nicely with a particular IDE is hit or miss, since I am not using any sophisticated IDE; and I imagine it may even be hit or miss for you with respect to some other IDE than the one you are using (I don't know how equivalent they are or are not). But anyhow if you want to leave `Dependencies<'add', T>` alone and rather than use aliases in `Dependencies` have another type operator `DependencyWithParams<'add', [T, RealType<T>]>` for the cases where one needs an implementation on `add` with a specific parameter tuple other than the "usual" one, that's fine with me. It should be pretty clear how to implement it in the existing framework of 4.6, or I am happy to take a stab if you like. Given that `Dependencies` is playing nice with your IDE, I can imagine that an analogous `DependencyWithParams` (feel to use a better name if you can think of one) implemented along the same lines would work fine, too. A slightly odd thing if we go this route is that then defining interfaces and implementations will be a bit less parallel with using them as dependencies. I don't see any way around using another identifier like `addReal` and an `AliasOf` type operator to include more interfaces for the `add` operation. So if we have to define aliases to create implementations, why not use them when declaring dependencies to indicate that it's this other interface we want? So my vote is for leaving 4.6 as it is (likely changing the keys for the additional instances of 'add' in Complex sqrt to make them a little clearer). That said, there's no real problem other than enlarging the API with having something like `DependencyWithParams<>` too. I just don't see a way to do something similar on the defining side, and I worry slightly that it might become confusing when to use `Dependencies<'foo', Type>`, when to use `DependencyWithParams<'foo', [TypeA, TypeB]>`, and when to use `Implementation<'foo', (c: TypeC, d: TypeD) => RetType>` (the notation we added in the conversation above to request input and output conversions -- we have not actually used in practice yet, but if we want to support it, we should find a real use case in which it would truly be the best way to do things, and include that in the prototype as soon as possible. I am fine with it in principle as we planned but have not yet thought of an actual instance in which it would be used...).
Collaborator

Sorry for the delay, I was sick.

I've briefly gone though the current set of functions of mathjs. Some observations:

  1. Most functions are either a function with a clear API that can be implemented for different datatypes (like add, multiply, sin, equal etc), or a generic function which (normally) only has a single implementation and are free to define their own syntax (like mean, std, intersect, eigs). So that will work out perfectly.

  2. There is a number of functions that do have an optional second argument. We can either decide that the official API requires this optional argument, or implement two versions of the function with aliasing. Examples:

    math.cbrt(x)
    math.cbrt(x, allRoots)
    
    math.floor(x)
    math.floor(x, n)
    
    math.log(x)
    math.log(x, base)
    
    math.eigs(x)
    math.eigs(x, precision)
    
  3. Besides the category of functions mentioned in (2), I do not see many places where we would need aliasing, so I think it will work out ok!

there's no real problem other than enlarging the API with having something like DependencyWithParams<> too.

Agree, it's good to have that possibility. And agree not to implement that right now.

So my vote is for leaving 4.6 as it is

Yes, agree 👍 (assuming that we can indeed make all of this work for real with typescript-rtti, including the AliasOf).

Sorry for the delay, I was sick. I've briefly gone though the current set of functions of mathjs. Some observations: 1. Most functions are either a function with a clear API that can be implemented for different datatypes (like `add`, `multiply`, `sin`, `equal` etc), _or_ a generic function which (normally) only has a single implementation and are free to define their own syntax (like `mean`, `std`, `intersect`, `eigs`). So that will work out perfectly. 2. There is a number of functions that do have an optional second argument. We can either decide that the official API requires this optional argument, or implement two versions of the function with aliasing. Examples: ```js math.cbrt(x) math.cbrt(x, allRoots) math.floor(x) math.floor(x, n) math.log(x) math.log(x, base) math.eigs(x) math.eigs(x, precision) ``` 3. Besides the category of functions mentioned in (2), I do not see many places where we would need aliasing, so I think it will work out ok! > there's no real problem other than enlarging the API with having something like `DependencyWithParams<>` too. Agree, it's good to have that possibility. And agree not to implement that right now. > So my vote is for leaving 4.6 as it is Yes, agree 👍 (assuming that we can indeed make all of this work for real with typescript-rtti, including the `AliasOf`).
Collaborator

@glen what is your opinion on how to implement functions that have multiple signatures, like eigs and say format?

math.eigs(x)
math.eigs(x, precision)

math.mean(a, b, c, ...)
math.mean(A)
math.mean(A, dim)

math.format(value)
math.format(value, options)
math.format(value, precision)
math.format(value, callback)

Do we implement each of them with a unique name and use AliasOf here?

@glen what is your opinion on how to implement functions that have multiple signatures, like `eigs` and say `format`? ``` math.eigs(x) math.eigs(x, precision) math.mean(a, b, c, ...) math.mean(A) math.mean(A, dim) math.format(value) math.format(value, options) math.format(value, precision) math.format(value, callback) ``` Do we implement each of them with a unique name and use AliasOf here?
Author
Owner

Responding to your latest two messages (sorry to hear about your illness):

  1. ...functions that do have an optional second argument

To avoid a lot of aliases just for this sake, since typescript perfectly well allows default argument values making the arguments optional, perhaps the new dispatcher should embrace them as well?

functions that have multiple signatures

(Like mean() of a list of scalars vs. mean() of an array, or format() which has numerous possible additional arguments, the semantics of which are determined by their type?)

I don't know, I think we are going to have to take these on something of a case-by-case basis. I have three observations, but I don't think they cover everything we are going to encounter.

A) the overarching interfaces in the interface/ directory are primarily for the sake of declaring dependencies, really. But a lot of the more specialized functions are never used as a dependency in anything else, or perhaps just one of their signatures is used as a dependency. Perhaps we should allow implementations that are "off-label", so to speak -- i.e. don't correspond to any overarching interface. We would just need some way of specifying what operation the implementations are for, and then it seems like the dispatcher could incorporate such implementations among the "standard" ones without any difficulty. It just would be trickier to use them as a dependency in something else. But if they aren't needed as dependencies, maybe that's not a big deal? Of course, they might later be needed as a dependency, and then someone would have to go back and add an alias in interfaces/ and change the implementation to refer to the alias, etc., which might not be a very nice process.

B) On mean() in particular, the function that takes a list of scalars to their mean versus the one that takes an array and an optional dimension to the array of means along that dimension feel like different functions to me anyway -- it's just a convenience for the client to be able to call them under the same name, so yes, for those I'd be totally fine having two interfaces, maybe even meanScalars and meanArray both aliased to mean (since it's really not clear which is the "primary" one).

C) On format(), since this is typescript, we are going to have to explicitly define the FormatOptions type anyway, and although the prototype has not yet dealt with implicit conversions, it seems to me that we can just define implicit conversions from number to FormatOptions (fills in the precision, everything else default values) and from a function to FormatOptions (fills in the callback, everything else default values). That should handle the apparent variation in types of the second argument. And so I think format has just one signature:

interface Signatures<T> {
   format: (item: T, options: FormatOptions) => string
}

and we would implement it type-by-type for T; in other words, a separate implementation of format in each subdirectory, one when T is number, one for bigint, one for Complex<T> that would of course have Dependencies<'format', T>, etc. This will be cleaner than the current implementation of format, which is not so easily extensible because there is a function in the utils directory that has to discriminate among all of the possible mathjs types, so any time a new type is added that function has to be extended to handle it.

I don't doubt there will be some sticky cases though that we have not yet worked through. I am thinking in particular of simplify, where you have to supply the expression of course, but then you can also specify any or all of the rules, scope, and options, more or less in any order with any included or not, and it just works out which is which from the types. That won't be straightforward to do in approach4.6, unfortunately. So it seems like we will either have to: (a) have one "full" interface that can be used as a dependency in other functions (simplify is definitely used elsewhere internally) and rely on proposal (A) I just made to supply the other ones to the client for convenience, or (b) have a whole bunch of aliases, which will be pretty ugly.

Also, not having worked through it in detail yet, I am worried about defining and using a dependency on "addMany" that takes 3 or more arguments of whatever types and adds them all up; it will have to rely on conversions, that we don't have implemented yet, so that it can promote numbers to Complex<number> as needed, etc. I have to admit it's unclear to me how that's all going to work with TypeScript.

Your thoughts? Are these concerns enough to steer us away from approach4.6 and back to one without single overarching interfaces for operations, like approach2.5? (That approach might need a slight syntax revison to approach2.6, since I think 2.5 has a "naming convention" rather than explicit aliasing.) I think such problems would not come up in something along those lines, at the cost of having to give the full list of parameter types rather than just a single type whenever specifying a dependency.

Thanks for the discussion.

Responding to your latest two messages (sorry to hear about your illness): > 2. ...functions that do have an optional second argument To avoid a lot of aliases just for this sake, since typescript perfectly well allows default argument values making the arguments optional, perhaps the new dispatcher should embrace them as well? > functions that have multiple signatures (Like `mean()` of a list of scalars vs. `mean()` of an array, or `format()` which has numerous possible additional arguments, the semantics of which are determined by their type?) I don't know, I think we are going to have to take these on something of a case-by-case basis. I have three observations, but I don't think they cover everything we are going to encounter. A) the overarching interfaces in the interface/ directory are primarily for the sake of declaring dependencies, really. But a lot of the more specialized functions are never used as a dependency in anything else, or perhaps just one of their signatures is used as a dependency. Perhaps we should allow _implementations_ that are "off-label", so to speak -- i.e. don't correspond to any overarching interface. We would just need some way of specifying what operation the implementations are for, and then it seems like the dispatcher could incorporate such implementations among the "standard" ones without any difficulty. It just would be trickier to use them as a dependency in something else. But if they aren't needed as dependencies, maybe that's not a big deal? Of course, they might later be needed as a dependency, and then someone would have to go back and add an alias in interfaces/ and change the implementation to refer to the alias, etc., which might not be a very nice process. B) On mean() in particular, the function that takes a list of scalars to their mean versus the one that takes an array and an optional dimension to the array of means along that dimension feel like different functions to me anyway -- it's just a convenience for the client to be able to call them under the same name, so yes, for those I'd be totally fine having two interfaces, maybe even meanScalars and meanArray _both_ aliased to mean (since it's really not clear which is the "primary" one). C) On format(), since this is typescript, we are going to have to explicitly define the FormatOptions type anyway, and although the prototype has not yet dealt with implicit conversions, it seems to me that we can just define implicit conversions from number to FormatOptions (fills in the precision, everything else default values) and from a function to FormatOptions (fills in the callback, everything else default values). That should handle the apparent variation in types of the second argument. And so I think format has just one signature: ``` interface Signatures<T> { format: (item: T, options: FormatOptions) => string } ``` and we would implement it type-by-type for T; in other words, a separate implementation of format in each subdirectory, one when T is number, one for bigint, one for `Complex<T>` that would of course have `Dependencies<'format', T>`, etc. This will be cleaner than the current implementation of format, which is not so easily extensible because there is a function in the utils directory that has to discriminate among all of the possible mathjs types, so any time a new type is added that function has to be extended to handle it. I don't doubt there will be some sticky cases though that we have not yet worked through. I am thinking in particular of `simplify`, where you have to supply the expression of course, but then you can also specify any or all of the rules, scope, and options, more or less in any order with any included or not, and it just works out which is which from the types. That won't be straightforward to do in approach4.6, unfortunately. So it seems like we will either have to: (a) have one "full" interface that can be used as a dependency in other functions (simplify is definitely used elsewhere internally) and rely on proposal (A) I just made to supply the other ones to the client for convenience, or (b) have a whole bunch of aliases, which will be pretty ugly. Also, not having worked through it in detail yet, I am worried about defining and using a dependency on "addMany" that takes 3 or more arguments of whatever types and adds them all up; it will have to rely on conversions, that we don't have implemented yet, so that it can promote numbers to `Complex<number>` as needed, etc. I have to admit it's unclear to me how that's all going to work with TypeScript. Your thoughts? Are these concerns enough to steer us away from approach4.6 and back to one without single overarching interfaces for operations, like approach2.5? (That approach might need a slight syntax revison to approach2.6, since I think 2.5 has a "naming convention" rather than explicit aliasing.) I think such problems would not come up in something along those lines, at the cost of having to give the full list of parameter types rather than just a single type whenever specifying a dependency. Thanks for the discussion.
Author
Owner

Thinking about these questions made me realize another point about some basic operations like say multiply(): as we scale up to the current full mathjs and beyond, we are going to need other interfaces besides the current ones. For matrices, we need to be able to multiply a scalar times a matrix, probably in either order, and mathjs needs to be able to handle a number times Matrix<Complex<number>> as well as a Matrix<number> times a Complex<number>, etc. Some of those possibilities will be handled by automatic conversions, we hope, but at least, it seems we will need to have two aliases

multiplyScalarMatrix: (s: T, m: Matrix<T>) => Matrix<T> and

multiplyMatrixScalar: (m: Matrix<T>, s: T) => Matrix<T>.

And now if any other collection type is added, will we have to add yet more aliases? Maybe the answer is yes, but maybe even if so it's not so bad because like in the case of Complex needing the addReal alias, the aliases can just be added in the Matrix subdirectory, etc., keeping things modular.

Or, is our apparent fate that the number of aliases for basic operations looks like it might balloon an indication that we should go back to an evolution of the approach2.5 branch, where there was no attempt to have an overarching type for each operation, at the cost of writing out the full list of parameter types whenever we declare a dependency? I mean, if we are going to have to write Dependencies<'multiplyScalarMatrix', T> anyway, is Dependencies<'multiply', [T, Matrix<T>]> really any worse? Or is the fact that then many instances of Dependencies<'multiply', T> would have to change to Dependencies<'multiply', [T, T]> if we abandon overarching interfaces too big a loss?

Or maybe we should go for a hybrid, where there is always exactly one "standard" interface for an operation, so that we can write something like DependStd<'add', T> and when you need any other interface you write out the parameters with Dependencies<'add', [Complex<T>, RealType<T>]>? And then there would be no aliases at the interface level, just a way to specify what operation a given implementation is for if it is not an instance of the standard interface? I will say my intuition is that in the long run a hybrid along these lines will end up smoother/cleaner than a proliferation of aliases that I am worried will seem ad-hoc and therefore become confusing...

Looking forward to your thoughts.

Thinking about these questions made me realize another point about some basic operations like say multiply(): as we scale up to the current full mathjs and beyond, we are going to need other interfaces besides the current ones. For matrices, we need to be able to multiply a scalar times a matrix, probably in either order, and mathjs needs to be able to handle a number times `Matrix<Complex<number>>` as well as a `Matrix<number>` times a `Complex<number>`, etc. Some of those possibilities will be handled by automatic conversions, we hope, but at least, it seems we will need to have two aliases `multiplyScalarMatrix: (s: T, m: Matrix<T>) => Matrix<T>` and `multiplyMatrixScalar: (m: Matrix<T>, s: T) => Matrix<T>`. And now if any other collection type is added, will we have to add yet more aliases? Maybe the answer is yes, but maybe even if so it's not **so** bad because like in the case of Complex needing the `addReal` alias, the aliases can just be added in the Matrix subdirectory, etc., keeping things modular. Or, is our apparent fate that the number of aliases for basic operations looks like it might balloon an indication that we should go back to an evolution of the approach2.5 branch, where there was no attempt to have an overarching type for each operation, at the cost of writing out the full list of parameter types whenever we declare a dependency? I mean, if we are going to have to write `Dependencies<'multiplyScalarMatrix', T>` anyway, is `Dependencies<'multiply', [T, Matrix<T>]>` really any worse? Or is the fact that then many instances of `Dependencies<'multiply', T>` would have to change to `Dependencies<'multiply', [T, T]>` if we abandon overarching interfaces too big a loss? Or maybe we should go for a hybrid, where there is always exactly one "standard" interface for an operation, so that we can write something like `DependStd<'add', T>` and when you need any other interface you write out the parameters with `Dependencies<'add', [Complex<T>, RealType<T>]>`? And then there would be no aliases at the interface level, just a way to specify what operation a given implementation is for if it is not an instance of the standard interface? I will say my intuition is that in the long run a hybrid along these lines will end up smoother/cleaner than a proliferation of aliases that I am worried will seem ad-hoc and therefore become confusing... Looking forward to your thoughts.
Collaborator

To avoid a lot of aliases just for this sake, since typescript perfectly well allows default argument values making the arguments optional, perhaps the new dispatcher should embrace them as well?

Yes, we can utilize default arguments I guess. I was mostly thinking about if we define say math.floor(x, n) as official API, we force everyone that wants to implement floor for a new datatype to support this optional n, wheras I expect that most common libraries only implement math.floor(x), so these functions cannot be imported as-is.

A) 👍
B) 👍
C) 👍 It makes indeed sense to split per data-type at least Let's see how it works out when we implement it.

I think we're fully on the same page. Let's see on a case-by-case basis whether to use aliases, resort to an "off-label" free-form dependency, and either have simple or full-fledged interfaces. Or some combination of that.

Let's see if we can avoid needing support for automatic conversions in dependencies, that would keep the system simpler I think. If really needed, we can think that through. I expect the challenge will not be on the TS side, but just inside the JS core that resolves/dispatches dependencies.

And now if any other collection type is added, will we have to add yet more aliases? Maybe the answer is yes, but maybe even if so it's not so bad because like in the case of Complex needing the addReal alias, the aliases can just be added in the Matrix subdirectory, etc., keeping things modular.
[...]
I mean, if we are going to have to write Dependencies<'multiplyScalarMatrix', T> anyway, is Dependencies<'multiply', [T, Matrix<T>]> really any worse?

Yes agree, I've been thinking about that too. It could be that the amount of aliases will grow out of hand. I'm not sure. One downside of the 2.5 approach to me is that you lose the guarentee that you're typing an existing interface. Not a show stopper, but having this guarentee is a nice added value to me.

Or maybe we should go for a hybrid, where there is always exactly one "standard" interface for an operation, so that we can write something like DependStd<'add', T> and when you need any other interface you write out the parameters with Dependencies<'add', [Complex<T>, RealType<T>]>?

I indeed think we will end up with some hybrid anyway: partly pre-defined interfaces that give you perfect guarentees and autocompletion etc, an a secondary API for the "special" cases. I'm not entirely sure whether Aliases will work out nicely in practice. I know for sure that your proposal will work and gives a clear, explainable API (you shared this idea before, and called it Dependencies and DependencyWithParams before, we can indeed think through a good name).

Thinking aloud: a third approach could be the approach4_typealias that we've discussed before, using plain and simple typealiases like FnAdd. That does not require a hybrid approach and gives a lot of freedom to define FnAddReal and any other "major" interfaces, depending on the need. The exact interface names are a less predictable and less DRY, and you do not have the correct function name in the interface though. We could add name information to the official definition, similar to the AliasOf approach:

// easy to use for everyone with an IDE with autocompletion ;) 
export type FnAdd<T> = Fn<'add', (a: T, b: T) => T>
export type FnAddReal<T, U> = Fn<'add', (a: T, b: U) => T>

// ...
export const add =
   <T>(dep: {
      add: FnAdd<T>,  // not DRY, but allows you to define your own aliases
      myAddReal: FnAddReal<T, U>
   }): FnAdd<Complex<T>> =>
      (w, z) => // ...

I don't like it that much though, I think I still prefer 4.6.

At this moment I think we should stick with approach 4.6, see if AliasOf works out in practice or not, and if not, go for the Dependencies and DependencyWithParams approach. Does that make sense?

> To avoid a lot of aliases just for this sake, since typescript perfectly well allows default argument values making the arguments optional, perhaps the new dispatcher should embrace them as well? Yes, we can utilize default arguments I guess. I was mostly thinking about if we define say `math.floor(x, n)` as official API, we force everyone that wants to implement `floor` for a new datatype to support this optional `n`, wheras I expect that most common libraries only implement `math.floor(x)`, so these functions cannot be imported as-is. A) 👍 B) 👍 C) 👍 It makes indeed sense to split per data-type at least Let's see how it works out when we implement it. I think we're fully on the same page. Let's see on a case-by-case basis whether to use aliases, resort to an "off-label" free-form dependency, and either have simple or full-fledged interfaces. Or some combination of that. Let's see if we can avoid needing support for automatic conversions in dependencies, that would keep the system simpler I think. If really needed, we can think that through. I expect the challenge will not be on the TS side, but just inside the JS core that resolves/dispatches dependencies. > And now if any other collection type is added, will we have to add yet more aliases? Maybe the answer is yes, but maybe even if so it's not **so** bad because like in the case of Complex needing the `addReal` alias, the aliases can just be added in the Matrix subdirectory, etc., keeping things modular. > [...] > I mean, if we are going to have to write `Dependencies<'multiplyScalarMatrix', T>` anyway, is `Dependencies<'multiply', [T, Matrix<T>]>` really any worse? Yes agree, I've been thinking about that too. It could be that the amount of aliases will grow out of hand. I'm not sure. One downside of the 2.5 approach to me is that you lose the guarentee that you're typing an existing interface. Not a show stopper, but having this guarentee is a nice added value to me. > Or maybe we should go for a hybrid, where there is always exactly one "standard" interface for an operation, so that we can write something like `DependStd<'add', T>` and when you need any other interface you write out the parameters with `Dependencies<'add', [Complex<T>, RealType<T>]>`? I indeed think we will end up with some hybrid anyway: partly pre-defined interfaces that give you perfect guarentees and autocompletion etc, an a secondary API for the "special" cases. I'm not entirely sure whether Aliases will work out nicely in practice. I know for sure that your proposal will work and gives a clear, explainable API (you shared this idea before, and called it `Dependencies` and `DependencyWithParams` before, we can indeed think through a good name). Thinking aloud: a third approach could be the `approach4_typealias` that we've discussed before, using plain and simple typealiases like `FnAdd`. That does not require a hybrid approach and gives a lot of freedom to define `FnAddReal` and any other "major" interfaces, depending on the need. The exact interface names are a less predictable and less DRY, and you do not have the correct function name in the interface though. We could add name information to the official definition, similar to the `AliasOf` approach: ```js // easy to use for everyone with an IDE with autocompletion ;) export type FnAdd<T> = Fn<'add', (a: T, b: T) => T> export type FnAddReal<T, U> = Fn<'add', (a: T, b: U) => T> // ... export const add = <T>(dep: { add: FnAdd<T>, // not DRY, but allows you to define your own aliases myAddReal: FnAddReal<T, U> }): FnAdd<Complex<T>> => (w, z) => // ... ``` I don't like it that much though, I think I still prefer 4.6. At this moment I think we should stick with approach 4.6, see if `AliasOf` works out in practice or not, and if not, go for the `Dependencies` and `DependencyWithParams` approach. Does that make sense?
Author
Owner

OK, so for now we will presume that there is one standard (often generic) interface for a given operation, to be defined in the interfaces/ directory, and that if certain types need other interfaces, such as addReal for add(Complex<T>, RealType<T>) or multiplyScalarVector for multiply(T, Vector<T>) and multiplyVectorScalar for multiply(Vector<T>, T), they will define them using AliasOf, and we will just see if the number of aliases and the complexity of finding the right alias to use for a dependency remains manageable? And if at some point the aliases become too out of, we will stick with a single primary interface, but switch from using aliases to a (possibly renamed) DependsWithParams('operation', [T1, T2]).

In other words, we are going to forge ahead with "approach4.6". As soon as I get a chance (I start teaching tomorrow) I will merge that branch into main here, and then whichever of us has the chance can start trying to get what we have so far to actually register functions correctly using typescript-rtti.

OK, so for now we will presume that there is one standard (often generic) interface for a given operation, to be defined in the `interfaces/` directory, and that if certain types need other interfaces, such as addReal for `add(Complex<T>, RealType<T>)` or multiplyScalarVector for `multiply(T, Vector<T>)` and multiplyVectorScalar for `multiply(Vector<T>, T)`, they will define them using AliasOf, and we will just see if the number of aliases and the complexity of finding the right alias to use for a dependency remains manageable? And if at some point the aliases become too out of, we will stick with a single primary interface, but switch from using aliases to a (possibly renamed) `DependsWithParams('operation', [T1, T2])`. In other words, we are going to forge ahead with "approach4.6". As soon as I get a chance (I start teaching tomorrow) I will merge that branch into main here, and then whichever of us has the chance can start trying to get what we have so far to actually register functions correctly using typescript-rtti.
Collaborator

💪 Yes, well summarized. Let's see how that goes.

I'll let you know when I start fiddling with typescript-rtti.

Have fun teaching!

💪 Yes, well summarized. Let's see how that goes. I'll let you know when I start fiddling with typescript-rtti. Have fun teaching!
glen closed this issue 2023-01-22 01:34:57 +00:00
Sign in to join this conversation.
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: glen/typocomath#6
No description provided.