With return types, specifications of implementations are too complicated #55

New Issue

Open

opened 2022-10-03 21:40:03 +00:00 by glen · 27 comments

glen commented 2022-10-03 21:40:03 +00:00

Owner

Copy Link

In a conversation with Jos de Jong today, the concern arose that with the return-type annotation (which is necessary for successful automated generation of TypeScript type definition files, and which facilitates more extensive injection of type-specific implementations to avoid re-dispatch), the specifications of fairly simple functions are becoming rather complicated and somewhat hard to read. Let's take a look at two examples, comparing two numbers and finding the square of the absolute value of a complex number:

export const compare = {
  'number,number': ({
    config
  }) => Returns(
    'NumInt', (x,y) => nearlyEqual(x, y, config.epsilon) ? 0 : (x > y ? 1 : -1))
}

(note nearlyEqual is a local helper, not exposed and not a dependency) and

export const absquare = {
   'Complex<T>': ({
      add,
      'self(T)': absq
   }) => {
      const midType = returnTypeOf(absq)
      const addImp = add.fromInstance.resolve(
         'add', `${midType},${midType}`, add)
      return Returns(
         returnTypeOf(addImp), z => addImp(absq(z.re), absq(z.im)))
   }
}

Now, to fully specify an implementation, we need to know the parameter types it works on, its dependencies, its return type (which could depend on the dependencies), and of course its implementing code. That's all there is in the compare example; there's no extraneous information, and so if it looks overly complicated, all that could possibly mean is that a more readable sytax for specifying those things is needed.

On the other hand, what's all that stuff going on in absquare for something that's really just the sum of the squares of the real and complex parts?? The difficulty here is that Pocomath's type language is not expressive enough to easily determine the result type of absquare, as well as to select the implementation needed for the intermediate function 'add', so it has to request all the implementations for add and select the correct one (pleasantly, still just once when the absquare function is generated for a type T, not every time the resulting absquare is called).

So for the first, I have the sense that Jos might prefer a syntax like:

export const compare = {
   'number, number => NumInt': ({config}) =>
      (x,y) => nearlyEqual(x, y, config.epsilon) ? 0 : (x > y ? 1 : -1))
}

I've objected to this in the past as it's the argument types that are unique among implementations, so they make a natural object key; but looking at this, I have to agree that the layout is better, so we can and should just make the infrastructure parse argType, argType, argType => retType and then enforce that we don't have two specifications for the same arg tuples.

As for the latter case, by a suggestion that's already in comments of the code, we could allow expressions like:

export const absquare = {
   'Complex<T> => V': ({
   	  'self(T): U' : absq,
      'add(U, U): V': plus
   }) => z => plus(absq(z.re), absq(z.im))
}

which would say that we need the implementation of add that can add two items of the return type of self on T, and that the return type of absquare on a Complex<T> will then be the return type of that implementation of add.

The main thing that worries me about this latter option is there's essentially nothing in that whole specification that TypeScript couldn't figure out from

function absquare<T>(z: Complex<T>) {
   return plus(absq(re(z)), absq(im(z))
}

except that TypeScript doesn't do selection of implementations of function based on the argument type. (Frankly, that's really all that's missing, so I have even contemplated writing a transformer for TypeScript that would allow implementations to be selected by argument type, like the one that gives type reflection at runtime that I used in typomath -- I think it's conceivably doable. But I also think it would mean clients of mathts would have to use that transformer, too, which makes that a nonstarter.)

So this line of thinking begs the questions:

(a) Well then if we are concerned about readability/simplicity, why wouldn't we just go whole hog and allow implementations to be specified by strings like

absquare(z: Complex<T>) = add(absquare(re(z)), absquare(im(z)))

since that has all the information that's actually needed?

(b) Well, but then what the heck would we be doing implementing a new language just for the sake of writing mathjs, one a lot like TypeScript but with "compile-time dispatch", more or less?

(c) So can we really, really not rewrite mathjs in TypeScript? (I think the answer to this is we really can't, because there's just no good way to write a function in terms of dispatching itself to other argument types like absquare does on complex types.)

(d) So should we just use another CAS instead of mathjs, rather than rewriting it and essentially implementing another language internally as a side effect? :-(

(e) But wait, mathjs already has a parser for another language built into it. And the last expression for absquare of a Complex<T> above looks quite a bit like a mathjs definition of a function, except for the presence of type information:

absquare(z) = add(absquare(re(z)), absquare(im(z)))

already parses perfectly well in mathjs's expression language. (It of course loops infinitely if you try to call it, because right now it doesn't know about different implementations for different types.) So could we possibly leverage the existing parser and by adding type specifications to it, actually write as much as possible of the "standard library" in the mathjs expression language, where it will infer return types and dependencies for us, and only implement (a) the lowest-level layer that maybe doesn't have dependencies anyway) and (b) the functions with complicated function bodies with things like loops that would definitely not be candidates for implementation in the mathjs expression language?

In a conversation with Jos de Jong today, the concern arose that with the return-type annotation (which is necessary for successful automated generation of TypeScript type definition files, and which facilitates more extensive injection of type-specific implementations to avoid re-dispatch), the specifications of fairly simple functions are becoming rather complicated and somewhat hard to read. Let's take a look at two examples, comparing two numbers and finding the square of the absolute value of a complex number: ``` export const compare = { 'number,number': ({ config }) => Returns( 'NumInt', (x,y) => nearlyEqual(x, y, config.epsilon) ? 0 : (x > y ? 1 : -1)) } ``` (note nearlyEqual is a local helper, not exposed and not a dependency) and ``` export const absquare = { 'Complex<T>': ({ add, 'self(T)': absq }) => { const midType = returnTypeOf(absq) const addImp = add.fromInstance.resolve( 'add', `${midType},${midType}`, add) return Returns( returnTypeOf(addImp), z => addImp(absq(z.re), absq(z.im))) } } ``` Now, to fully specify an implementation, we need to know the parameter types it works on, its dependencies, its return type (which could depend on the dependencies), and of course its implementing code. That's all there is in the compare example; there's no extraneous information, and so if it looks overly complicated, all that could possibly mean is that a more readable sytax for specifying those things is needed. On the other hand, what's all that stuff going on in absquare for something that's really just the sum of the squares of the real and complex parts?? The difficulty here is that Pocomath's type language is not expressive enough to easily determine the result type of absquare, as well as to select the implementation needed for the intermediate function 'add', so it has to request all the implementations for add and select the correct one (pleasantly, still just once when the absquare function is generated for a type T, not every time the resulting absquare is called). So for the first, I have the sense that Jos might prefer a syntax like: ``` export const compare = { 'number, number => NumInt': ({config}) => (x,y) => nearlyEqual(x, y, config.epsilon) ? 0 : (x > y ? 1 : -1)) } ``` I've objected to this in the past as it's the argument types that are unique among implementations, so they make a natural object key; but looking at this, I have to agree that the layout is better, so we can and should just make the infrastructure parse `argType, argType, argType => retType` and then enforce that we don't have two specifications for the same arg tuples. As for the latter case, by a suggestion that's already in comments of the code, we could allow expressions like: ``` export const absquare = { 'Complex<T> => V': ({ 'self(T): U' : absq, 'add(U, U): V': plus }) => z => plus(absq(z.re), absq(z.im)) } ``` which would say that we need the implementation of `add` that can add two items of the return type of self on T, and that the return type of absquare on a `Complex<T>` will then be the return type of that implementation of `add`. The main thing that worries me about this latter option is there's essentially nothing in that whole specification that TypeScript couldn't figure out from ``` function absquare<T>(z: Complex<T>) { return plus(absq(re(z)), absq(im(z)) } ``` except that TypeScript doesn't do selection of implementations of function based on the argument type. (Frankly, that's really all that's missing, so I have even contemplated writing a transformer for TypeScript that would allow implementations to be selected by argument type, like the one that gives type reflection at runtime that I used in typomath -- I think it's conceivably doable. But I also think it would mean _clients_ of mathts would have to use that transformer, too, which makes that a nonstarter.) So this line of thinking begs the questions: (a) Well then if we are concerned about readability/simplicity, why wouldn't we just go whole hog and allow implementations to be specified by strings like `absquare(z: Complex<T>) = add(absquare(re(z)), absquare(im(z)))` since that has all the information that's actually needed? (b) Well, but then what the heck would we be doing implementing a new language just for the sake of writing mathjs, one a lot like TypeScript but with "compile-time dispatch", more or less? (c) So can we _really, really_ not rewrite mathjs in TypeScript? (I _think_ the answer to this is we _really_ can't, because there's just no good way to write a function in terms of dispatching itself to other argument types like absquare does on complex types.) (d) So should we just use another CAS _instead_ of mathjs, rather than rewriting it and essentially implementing another language internally as a side effect? :-( (e) But wait, mathjs _already_ has a parser for another language built into it. And the last expression for absquare of a `Complex<T>` above looks **quite** a bit like a mathjs definition of a function, except for the presence of type information: `absquare(z) = add(absquare(re(z)), absquare(im(z)))` already parses perfectly well in mathjs's expression language. (It of course loops infinitely if you try to call it, because right now it doesn't know about different implementations for different types.) So could we possibly leverage the existing parser and by adding type specifications to it, actually write as much as possible of the "standard library" in the mathjs expression language, where it will infer return types and dependencies for us, and only implement (a) the lowest-level layer that maybe doesn't have dependencies anyway) and (b) the functions with complicated function bodies with things like loops that would definitely not be candidates for implementation in the mathjs expression language?

glen commented 2022-10-04 01:59:07 +00:00

Author

Owner

Copy Link

I think the final question in the issue statement above should be taken seriously: would it be possible to blur the line between implementing operators in JavaScript and defining them in the mathjs expression language (MEL?), or at least move them closer together, so that it really does make sense to express noticeable portions of the standard library in the mathjs expression language? In particular, if MEL is beefed up just a little to have type markings and if it is then able to add implmentations of operators to the math object, I don't see why for functions with simple bodies there's any reason why the code/performance produced by adding an implementation with a mathjs expression should be any worse than adding it with JavaScript code. After all, it compiles down to a JavaScript function, and I think if we are careful it should boil down to pretty much the same JavaScript function as we would write by hand. This of course will require some thought and testing to make sure there really is not much or any cruft left in the resulting JavaScript function as a result of the compilation process.

My biggest worry in this process is the transform that shifts indices to be one-based. Presumably that has an operational cost that we won't want to pay in the implementations added via MEL. Not really sure exactly how that works in detail or if it's really a worry.

I think the final question in the issue statement above should be taken seriously: would it be possible to blur the line between implementing operators in JavaScript and defining them in the mathjs expression language (MEL?), or at least move them closer together, so that it really does make sense to express noticeable portions of the standard library in the mathjs expression language? In particular, if MEL is beefed up just a little to have type markings and if it is then able to add implmentations of operators to the math object, I don't see why for functions with simple bodies there's any reason why the code/performance produced by adding an implementation with a mathjs expression should be any worse than adding it with JavaScript code. After all, it compiles down to a JavaScript function, and I think if we are careful it should boil down to pretty much the same JavaScript function as we would write by hand. This of course will require some thought and testing to make sure there really is not much or any cruft left in the resulting JavaScript function as a result of the compilation process. My biggest worry in this process is the transform that shifts indices to be one-based. Presumably that has an operational cost that we won't want to pay in the implementations added via MEL. Not really sure exactly how that works in detail or if it's really a worry.

josdejong commented 2022-10-05 10:13:55 +00:00

Copy Link

Yes, interesting thoughts.

it's the argument types that are unique among implementations

This is indeed true, but I think pocomath can just guard against this and throw an error when there is a duplicate, also when it first has to extract the arguments part from the signature key.

About (a) to (e): Wow. That is thinking out of the box. It can probably be useful for simple cases. But it feels to me we should not try to write a full language for this and just use JavaScript. It is an interesting idea to see if we can leverage the expression parser to write these kind of simple functions. I have the feeling this could be an addition on top of pocomath, but not essential to think through for this API discussion?

Yes, interesting thoughts. > it's the argument types that are unique among implementations This is indeed true, but I think pocomath can just guard against this and throw an error when there is a duplicate, also when it first has to extract the arguments part from the signature key. About (a) to (e): Wow. That is thinking out of the box. It can probably be useful for simple cases. But it feels to me we should not try to write a full language for this and just use JavaScript. It is an interesting idea to see if we can leverage the expression parser to write these kind of simple functions. I have the feeling this could be an addition _on top of_ pocomath, but not essential to think through for this API discussion?

josdejong commented 2022-10-05 10:14:15 +00:00

Copy Link

So currently we can have up to four nestings I think:

1. an object with function names as key, and signatures as value (optional)
2. and object with arguments as key, and a factory function as value
3. a factory function with an object with dependencies as argument, and the function implementation as return type
4. a Returns callback with the return type and a callback as arguments (optional)

Some ideas:

• less nesting is most likely easier to read/understand/use
• it would be nice if syntax for signatures is the same as the notation for dependencies.
• it would be nice if the syntax is the same as TypeScript.
• it would be nice if the API would be layered, where the common use case is very simple and flat, and only advanced use cases like a return type depending on config requires more nesting.

Thinking aloud about an API, here some thoughts:

(a) current implementation: very concise, but quite some nesting

export const add = {'bigint,bigint': () => Returns('bigint', (a,b) => a + b)}

(b) we could allow (optional?) named notations with the same syntax as the dependencies and TypeScript, and with the same return type notation as TypeScript. Putting the name in the signature also allows putting multiple functions in one object (instead of needing an extra nesting for that):

export const myFns = {
  'add(bigint, bigint) : bigint': () => (a,b) => a + b,
  'negate(bigint) : bigint': () => (a) => -a,
  'subtract(bigint, bigint) : bigint': resolve(({
    'add(bigint, bigint)': add,
    'negate(bigint)': negate
  }) => (x,y) => add(x, negate(y)))
}

(c) we could make providing a factory function for the dependencies optional when we wrap the factory function in a named function like resolve (similar to the referTo syntax of typed-function). That makese the API more explicit and self explanatory:

export const myFns = {
  'add(bigint, bigint) : bigint': (a,b) => a + b,
  'negate(bigint) : bigint': (a) => -a,
  'subtract(bigint, bigint) : bigint': resolve(({
    'add(bigint, bigint)': add,
    'negate(bigint)': negate
  }) => (x,y) => add(x, negate(y)))
}

(d) I think it will be helpful if we write most of the code in TypeScript. That will result in quite some duplication of all the types. It feels not ideal but I think that will become the reality. Something like:

export const add = {
  'add(bigint, bigint) : bigint': () => (a: bigint, b: bigint) => a + b
}

(e) we could allow optionally naming the arguments, so you can copy/paste the signature from TypeScript. Maybe we can use all this information not only to generate a typed-function and generate TS type definitions, but also to gererate documentation on the fly? That would require attaching extra meta data like description and examples:

export const add = {
  'add(a: bigint, b: bigint) : bigint': document({
    description: '...',
    examples: [...],
    implementation: () => (a: bigint, b: bigint) => a + b
  })
}

So currently we can have up to four nestings I think: 1. an object with function names as key, and signatures as value (optional) 2. and object with arguments as key, and a factory function as value 3. a factory function with an object with dependencies as argument, and the function implementation as return type 4. a `Returns` callback with the return type and a callback as arguments (optional) Some ideas: - less nesting is most likely easier to read/understand/use - it would be nice if syntax for signatures is the same as the notation for dependencies. - it would be nice if the syntax is the same as TypeScript. - it would be nice if the API would be layered, where the common use case is very simple and flat, and only advanced use cases like a return type depending on config requires more nesting. Thinking aloud about an API, here some thoughts: (a) current implementation: very concise, but quite some nesting ```js export const add = {'bigint,bigint': () => Returns('bigint', (a,b) => a + b)} ``` (b) we could allow (optional?) named notations with the same syntax as the dependencies and TypeScript, and with the same return type notation as TypeScript. Putting the name in the signature also allows putting multiple functions in one object (instead of needing an extra nesting for that): ```js export const myFns = { 'add(bigint, bigint) : bigint': () => (a,b) => a + b, 'negate(bigint) : bigint': () => (a) => -a, 'subtract(bigint, bigint) : bigint': resolve(({ 'add(bigint, bigint)': add, 'negate(bigint)': negate }) => (x,y) => add(x, negate(y))) } ``` (c) we could make providing a factory function for the dependencies optional when we wrap the factory function in a named function like `resolve` (similar to the `referTo` syntax of `typed-function`). That makese the API more explicit and self explanatory: ```js export const myFns = { 'add(bigint, bigint) : bigint': (a,b) => a + b, 'negate(bigint) : bigint': (a) => -a, 'subtract(bigint, bigint) : bigint': resolve(({ 'add(bigint, bigint)': add, 'negate(bigint)': negate }) => (x,y) => add(x, negate(y))) } ``` (d) I think it will be helpful if we write most of the code in TypeScript. That will result in quite some duplication of all the types. It feels not ideal but I think that will become the reality. Something like: ```ts export const add = { 'add(bigint, bigint) : bigint': () => (a: bigint, b: bigint) => a + b } ``` (e) we could allow optionally naming the arguments, so you can copy/paste the signature from TypeScript. Maybe we can use all this information not only to generate a typed-function and generate TS type definitions, but also to gererate documentation on the fly? That would require attaching extra meta data like description and examples: ```ts export const add = { 'add(a: bigint, b: bigint) : bigint': document({ description: '...', examples: [...], implementation: () => (a: bigint, b: bigint) => a + b }) } ```

glen commented 2022-10-05 17:29:22 +00:00

Author

Owner

Copy Link

we should not try to write a full language for this and just use JavaScript. It is an interesting idea to see if we can leverage the expression parser to write these kind of simple functions. I have the feeling this could be an addition on top of pocomath, but not essential to think through for this API discussion?

Well, I completely agree: I have no interest in trying to write yet another language, that's for sure. We're already talking about JavaScript and TypeScript, and mathjs already has its own expression language that I think should have a name (it would make the docs clearer at points), I fancifully use MEL above but I think there could/should be a better name.

But there is an actual problem. For auto TS definition generation (and internal bypassing of re-dispatch), we need to track the argument types and return types of all typed-function implementations. And currently in Pocomath for the complex number implementation of the "square of the absolute value" function, this leads to the very complicated code of the second example of the original issue statement aboce. But that code is not expressing anything that isn't already captured in something like

absquare(z: Complex<T>) = add(absquare(re(z)), absquare(im(z)))

So I think to make a "Pocomath-style" refactoring of mathjs practical, we need to do something to get that first example at least closer to this brief thing. The intermediate "decent" version I proposed

export const absquare = {
   'Complex<T> => V': ({
      'self(T): U' : absq,
      'add(U, U): V': plus
   }) => z => plus(absq(z.re), absq(z.im))
}

is still fairly cumbersome/redundant AND is already a lot of work to implement over the curent Pocomath prototype, quite possibly more work than adding type annotations to the current mathjs internal expression language. So I think how we may want to specify functions like absquare in the library is a real concern and one without a settled answer. I am being serious when I say that implementing as much of the mathjs "standard library" in its own expression language actually seems entirely sensible, if it can be done without performance penalty.

> > we should not try to write a full language for this and just use JavaScript. It is an interesting idea to see if we can leverage the expression parser to write these kind of simple functions. I have the feeling this could be an addition _on top of_ pocomath, but not essential to think through for this API discussion? > Well, I completely agree: I have no interest in trying to write yet another language, that's for sure. We're already talking about JavaScript and TypeScript, and mathjs already has its own expression language that I think should have a name (it would make the docs clearer at points), I fancifully use MEL above but I think there could/should be a better name. But there is an actual problem. For auto TS definition generation (and internal bypassing of re-dispatch), we need to track the argument types and return types of all typed-function implementations. And currently in Pocomath for the complex number implementation of the "square of the absolute value" function, this leads to the very complicated code of the second example of the original issue statement aboce. But that code is not expressing anything that isn't already captured in something like `absquare(z: Complex<T>) = add(absquare(re(z)), absquare(im(z)))` So I think to make a "Pocomath-style" refactoring of mathjs practical, we need to do something to get that first example at least closer to this brief thing. The intermediate "decent" version I proposed ``` export const absquare = { 'Complex<T> => V': ({ 'self(T): U' : absq, 'add(U, U): V': plus }) => z => plus(absq(z.re), absq(z.im)) } ``` is still fairly cumbersome/redundant AND is already a lot of work to implement over the curent Pocomath prototype, quite possibly more work than adding type annotations to the current mathjs internal expression language. So I think how we may want to specify functions like absquare in the library is a real concern and one without a settled answer. I am being serious when I say that implementing as much of the mathjs "standard library" in its own expression language actually seems entirely sensible, if it can be done without performance penalty.

glen commented 2022-10-05 18:35:37 +00:00

Author

Owner

Copy Link

Now on to your (a) to (e). :-)

(a) Yes, readability would be better with the return type moved alongside the argument types. But we need to decide how to handle situations like sqrt in which the return type depends on the dependencies. I.e., if predictable is true, then the return type of the sqrt of a number is number, if not it is number|Complex<number>. How is this to be captured? Do we want an "escape" for such cases, something like

'sqrt(n: number): SQRT_TYPE' : ({config}) => ({
   SQRT_TYPE: config.predictable ? 'number' : 'number|Complex<number>',
   implementation: config.predictable ? (n => math.sqrt(n)) : (n => generalSqrt(n)
})

Or is there a better scheme for these instances? I don't think the answer is to get rid of config.predictable, I think there are other instances in which the return type is affected by the details of the dependencies.

(b) I do think it could work to move the function name into the key, to unify it with the syntax for the dependencies. That's a nice symmetry. But I don't see how this reduces nesting. Since exported identifiers have to be identifiers, this is just trading:

export add = {'(number, number): number' : () => (a, b) => a+b}

for

export fns = {'add(number, number): number': () => (a, b) => a+b}

Unless I am missing something here, I don't see any structural difference between these.

(c) Here I have a big concern: if you look at the current code for mathjs, having dependencies is far more common than not having dependencies. I'd say something like 95% of library functions have dependencies. So the syntax should be optimized for that case, not for the rare instance where there are no dependencies. That would suggest something like:

export fns = {
    'add(bigint, bigint): bigint': direct((a, b) => a+b)),
    'negate(bigint): bigint': direct(a => -a),
    'subtract(T, T): T' => ({'add(T,T)': add, 'negate(T)': negate}) =>
        (a, b) => add(a, negate(b))
}

i.e., it's the ones without dependencies that I have wrapped in a special call, not the ones with. That will lead to much less nesting overall. And in fact this direct can very easily be defined as function direct(f) { return () => f }. (And it could be called noDependencies or independent or base or ... as you prefer.)

Note that I replaced subtract(bigint, bigint) with subtract(T,T) to remind ourselves that we certainly want this sort of subtract (if we did it by negation and adding, which is probably not best in practice) to be generic so it doesn't have to be reiterated for each type.

(d) I am pretty uncomfortable with a design that will forever require implementors to state each type twice. If we were going to go that way, I would strongly recommend that we use a typescript run-time type information generator so that we could write something like just

add: direct((a:bigint, b:bigint) => a+b)

and the infrastructure would get the TypeScript type info from the supplied function and use that to supply the signature of this implementation to typed-function.

This approach now begs the question of whether in this "new mathjs" we want to limit ourselves to types that can be precisely captured in the TypeScript type system? typed-function currently has no such restriction; anything that can be detected at runtime can be a "type". There is actually currently a minor usage of this in mathjs: there some occurrenes the type identifier which is a string satisfying certain syntactic constraints (alphabetic followed by alphanumeric), but there is no way in TypeScript to create an exactly corresponding type. Similarly Pocomath showed a NumInt type as a proof of concept (not certain we actually want that in mathjs, but maybe) but there is no way to make an "integer instance of built-in number" type in TypeScript. Should this sort of thing be disallowed to ensure all individual implementations can be written in TypeScript? Is that the goal?

We also have to make sure this works for generics, but I think it is OK; I think we could write

const fns = {
    subtract: <T>(
        {add, negate}: {add: (a:T, b:T) => T, negate: (a:T) => T}) => 
        (a: T, b: T) => add(a, negate(b))
}

and we could extract the type information from the implementation and its destructuring parameter for the dependencies and figure out which implementations of add and negate we need to supply to an instance of subtract to create the corresponding implementaton of that. (But it isn't super pretty, I must say, and fairly annoying that TypeScript requires that the keys 'add' and 'negate' to be reiterated.)

(e) I am totally fine with regularizing and incorporating documentation and examples so long as it can be done without too much verboseness. Some concerns:

• the examples you give seem to have introduced a new redundancy between the identifier being exported and the beginning of the key of the exported object (both are 'add' in the example), that doesn't seem great.

• Where is the locus of documentation for a given operation? In other words, currently mathjs has each operation in its own file, containing the implementations for all types, so that's a clear place to put its documentation. If we do end up distributing implementations for 'add' into multiple files, where does the overall documentation for 'add' go? Or, if we put documentation on individual implmentations, like add for bigint and add for Complex<T>, etc., how would that be collected into a coherent overall documentation?

I like the idea of including examples, it seems those can easily be collected together, but it seems to me that the overall documentation for an operation needs to be in a single place. I like the idea of documentation being along with code, though, rather than in a new place away from any implementation. I mean, it could by convention go with the number implementation, except I do believe there are operations that do not apply to numbers at all, so that convention can't work for every operation... Documentation conventions are not something yet considered in the proof-of-concept, and do need to be worked out; I don't think there's an obvious answer, unfortunately.

Now on to _your_ (a) to (e). :-) (a) Yes, readability would be better with the return type moved alongside the argument types. But we need to decide how to handle situations like `sqrt` in which the return type depends on the dependencies. I.e., if `predictable` is true, then the return type of the sqrt of a number is number, if not it is `number|Complex<number>`. How is this to be captured? Do we want an "escape" for such cases, something like ``` 'sqrt(n: number): SQRT_TYPE' : ({config}) => ({ SQRT_TYPE: config.predictable ? 'number' : 'number|Complex<number>', implementation: config.predictable ? (n => math.sqrt(n)) : (n => generalSqrt(n) }) ``` Or is there a better scheme for these instances? I don't think the answer is to get rid of config.predictable, I think there are other instances in which the return type is affected by the details of the dependencies. (b) I do think it could work to move the function name into the key, to unify it with the syntax for the dependencies. That's a nice symmetry. But I don't see how this reduces nesting. Since exported identifiers have to be identifiers, this is just trading: ``` export add = {'(number, number): number' : () => (a, b) => a+b} ``` for ``` export fns = {'add(number, number): number': () => (a, b) => a+b} ``` Unless I am missing something here, I don't see any structural difference between these. (c) Here I have a big concern: if you look at the current code for mathjs, **having** dependencies is far more common than not having dependencies. I'd say something like 95% of library functions have dependencies. So the syntax should be optimized for that case, not for the rare instance where there are no dependencies. That would suggest something like: ``` export fns = { 'add(bigint, bigint): bigint': direct((a, b) => a+b)), 'negate(bigint): bigint': direct(a => -a), 'subtract(T, T): T' => ({'add(T,T)': add, 'negate(T)': negate}) => (a, b) => add(a, negate(b)) } ``` i.e., it's the ones _without_ dependencies that I have wrapped in a special call, not the ones _with_. That will lead to much _less_ nesting overall. And in fact this `direct` can very easily be defined as `function direct(f) { return () => f }`. (And it could be called `noDependencies` or `independent` or `base` or ... as you prefer.) Note that I replaced `subtract(bigint, bigint)` with `subtract(T,T)` to remind ourselves that we certainly want this sort of subtract (if we did it by negation and adding, which is probably not best in practice) to be generic so it doesn't have to be reiterated for each type. (d) I am pretty uncomfortable with a design that will forever require implementors to state each type twice. If we were going to go that way, I would strongly recommend that we use a typescript run-time type information generator so that we could write something like just `add: direct((a:bigint, b:bigint) => a+b)` and the infrastructure would get the TypeScript type info from the supplied function and use that to supply the signature of this implementation to typed-function. This approach now begs the question of whether in this "new mathjs" we want to limit ourselves to types that can be precisely captured in the TypeScript type system? typed-function currently has no such restriction; anything that can be detected at runtime can be a "type". There is actually currently a minor usage of this in mathjs: there some occurrenes the type `identifier` which is a string satisfying certain syntactic constraints (alphabetic followed by alphanumeric), but there is no way in TypeScript to create an exactly corresponding type. Similarly Pocomath showed a NumInt type as a proof of concept (not certain we actually want that in mathjs, but maybe) but there is no way to make an "integer instance of built-in number" type in TypeScript. Should this sort of thing be disallowed to ensure all individual implementations can be written in TypeScript? Is that the goal? We also have to make sure this works for generics, but I think it is OK; I think we could write ``` const fns = { subtract: <T>( {add, negate}: {add: (a:T, b:T) => T, negate: (a:T) => T}) => (a: T, b: T) => add(a, negate(b)) } ``` and we could extract the type information from the implementation and its destructuring parameter for the dependencies and figure out which implementations of add and negate we need to supply to an instance of subtract to create the corresponding implementaton of that. (But it isn't super pretty, I must say, and fairly annoying that TypeScript requires that the keys 'add' and 'negate' to be reiterated.) (e) I am totally fine with regularizing and incorporating documentation and examples so long as it can be done without too much verboseness. Some concerns: * the examples you give seem to have introduced a new redundancy between the identifier being exported and the beginning of the key of the exported object (both are 'add' in the example), that doesn't seem great. * Where is the locus of documentation for a given operation? In other words, currently mathjs has each operation in its own file, containing the implementations for all types, so that's a clear place to put its documentation. If we do end up distributing implementations for 'add' into multiple files, where does the overall documentation for 'add' go? Or, if we put documentation on individual implmentations, like add for bigint and add for `Complex<T>`, etc., how would that be collected into a coherent overall documentation? I like the idea of including examples, it seems those can easily be collected together, but it seems to me that the overall documentation for an operation needs to be in a single place. I like the idea of documentation being along with code, though, rather than in a new place away from any implementation. I mean, it could by convention go with the number implementation, except I do believe there are operations that do not apply to numbers at all, so that convention can't work for _every_ operation... Documentation conventions are not something yet considered in the proof-of-concept, and do need to be worked out; I don't think there's an obvious answer, unfortunately.

glen referenced this issue 2022-10-06 01:13:03 +00:00

Importing and combining Pocomath instances inconsistent #56

josdejong commented 2022-10-06 10:53:33 +00:00

Copy Link

(f)
Hmm. Yes I'm starting to understand the difficulties you mention regarding return types and TypeScript. Would it be even possible to determine the return types statically at all, when they depend on runtime configuration? Would it be acceptable if we would simply not support dynamic return types in the first place? It would make life so much easier. In that case we would be a bit conservative, like define sqrt to simply return number | Complex. Maybe that is... okish? You can always define a separate sqrt fumction that just works for numbers and use that, or accept having to cast the outcome in TS to number.

(g)

So I think how we may want to specify functions like absquare in the library is a real concern and one without a settled answer. I am being serious when I say that implementing as much of the mathjs "standard library" in its own expression language actually seems entirely sensible, if it can be done without performance penalty.

Yes, it's a really cool idea 😎, worth thinking through. To get an idea of how much use this will be, we will need to go through the list with current functions and identify how many of them could benefit from this compact notation.

(b) there is only a difference when you want to use a single object to define multiple functions. I'm not sure if that will be come a regular pattern. Right now in pocomath you mostly define functions one by one, each having their own object. Thinking about it, it is pobably better to define functions individually in general, that allows for tree shaking. So, in short, the only remaining reason to allow naming the signature is for symmetry and readability, and just having more flexibility. And, we can keep it optional.

(c) Hm, yeah, that's true, having dependencies is more common than not having them. But even despite being more verbose, seeing an explicit resolve(...) is appealing to me from a code-readability point of view (same discussion as we had with typed-function 😉 ). I expect that when we would show these three variants (anonymous arrow function, resolve(...), and direct(...)) to an unsuspecting, unknowing programmer and ask what it does, the resolve variant will be easiest to grasp.

(d)

I would strongly recommend that we use a typescript run-time type information generator so that we could write something like just add: direct((a:bigint, b:bigint) => a+b)

Yes that looks perfect. But that would require some special TypeScript setup, right? So that could be a barrier to use mathjs yourself. We haven't really worked out an experiment in that regard. It could be optional though, alongside "regular" signatures, and we could give the function a clear naming to explain what's going on. add: inferSignatureFromTS((a:bigint, b:bigint) => a+b). It could be a rabbithole though with tricky edge cases when converting TS type definitions into the typed-function syntax. I'm not sure if we should go that direction. But the alternative, having a lot of duplication, is also far from ideal.

(e) I have to give these documentation ideas a bit more thought, you have some good feedback there. It is not a blocker for the current discussion, we can implement something in this direction in a later stage. Shall we park this idea and focus on the core API for now?

**(f)** Hmm. Yes I'm starting to understand the difficulties you mention regarding return types and TypeScript. Would it be even possible to determine the return types statically at all, when they depend on runtime configuration? Would it be acceptable if we would simply not support dynamic return types in the first place? It would make life _so_ much easier. In that case we would be a bit conservative, like define `sqrt` to simply return `number | Complex`. Maybe that is... okish? You can always define a separate `sqrt` fumction that just works for numbers and use that, or accept having to cast the outcome in TS to `number`. **(g)** > So I think how we may want to specify functions like absquare in the library is a real concern and one without a settled answer. I am being serious when I say that implementing as much of the mathjs "standard library" in its own expression language actually seems entirely sensible, if it can be done without performance penalty. Yes, it's a really cool idea 😎, worth thinking through. To get an idea of how much use this will be, we will need to go through the list with current functions and identify how many of them could benefit from this compact notation. **(b)** there is only a difference when you want to use a single object to define multiple functions. I'm not sure if that will be come a regular pattern. Right now in pocomath you mostly define functions one by one, each having their own object. Thinking about it, it is pobably better to define functions individually in general, that allows for tree shaking. So, in short, the only remaining reason to allow naming the signature is for symmetry and readability, and just having more flexibility. And, we can keep it optional. **(c)** Hm, yeah, that's true, having dependencies is more common than not having them. But even despite being more verbose, seeing an explicit `resolve(...)` is appealing to me from a code-readability point of view (same discussion as we had with typed-function :wink: ). I expect that when we would show these three variants (anonymous arrow function, `resolve(...)`, and `direct(...)`) to an unsuspecting, unknowing programmer and ask what it does, the `resolve` variant will be easiest to grasp. **(d)** > I would strongly recommend that we use a typescript run-time type information generator so that we could write something like just `add: direct((a:bigint, b:bigint) => a+b)` Yes that looks perfect. But that would require some special TypeScript setup, right? So that could be a barrier to use mathjs yourself. We haven't really worked out an experiment in that regard. It could be optional though, alongside "regular" signatures, and we could give the function a clear naming to explain what's going on. `add: inferSignatureFromTS((a:bigint, b:bigint) => a+b)`. It could be a rabbithole though with tricky edge cases when converting TS type definitions into the typed-function syntax. I'm not sure if we should go that direction. But the alternative, having a lot of duplication, is also far from ideal. **(e)** I have to give these documentation ideas a bit more thought, you have some good feedback there. It is not a blocker for the current discussion, we can implement something in this direction in a later stage. Shall we park this idea and focus on the core API for now?

glen commented 2022-10-06 12:32:12 +00:00

Author

Owner

Copy Link

I am on my phone at the moment, so I am going to respond one comment per letter, sorry.

(f) Well Pocomath works totally fine with return types that depend on the configuration. As far as specification goes, it is no problem if you have an optional syntax where you are allowed to have the return type depend on the dependencies. As far as Typescript goes, the .d.ts generator will produce proper typings of a math object in a given state, so we run it for a handful of different configurations and give clients a choice, or they can set up an instance exactly the way they want and dump the definitions themselves. I don't think any of this is a reason not to have types affected by config, personally.

I am on my phone at the moment, so I am going to respond one comment per letter, sorry. (f) Well Pocomath works totally fine with return types that depend on the configuration. As far as specification goes, it is no problem if you have an optional syntax where you are allowed to have the return type depend on the dependencies. As far as Typescript goes, the .d.ts generator will produce proper typings of a math object in a given state, so we run it for a handful of different configurations and give clients a choice, or they can set up an instance exactly the way they want and dump the definitions themselves. I don't think any of this is a reason not to have types affected by config, personally.

glen commented 2022-10-06 12:42:26 +00:00

Author

Owner

Copy Link

(g) not 100% sure how to do that count. I mean the percentage is fairly large in pocomath but of course i did mostly the simpler operations. And i didn't try to match the algorithms of mathjs, I was deliberately writing from scratch to free up the space of possibilities. I suppose in the actual conversion we are going to generally try to leave algorithms alone unless there is clearly a much more efficient way to do some particular thing?

anyhow, it will be some significant work to get to a hybrid state where some stuff can be defined new style while many current source files remain exactly as they are. but I think from our conversation we need to get to such a state for the sake of transition. so we can just work toward that state with only javascript definitions first, and then when we encounter some implementations where it would be convenient to have mathjs expression implementations, see how much additional work that would be. my guess is not all that much, really.

(g) not 100% sure how to do that count. I mean the percentage is fairly large in pocomath but of course i did mostly the simpler operations. And i didn't try to match the algorithms of mathjs, I was deliberately writing from scratch to free up the space of possibilities. I suppose in the actual conversion we are going to generally try to leave algorithms alone unless there is clearly a much more efficient way to do some particular thing? anyhow, it will be some significant work to get to a hybrid state where some stuff can be defined new style while many current source files remain exactly as they are. but I think from our conversation we need to get to such a state for the sake of transition. so we can just work toward that state with only javascript definitions first, and then when we encounter some implementations where it would be convenient to have mathjs expression implementations, see how much additional work that would be. my guess is not all that much, really.

glen commented 2022-10-06 12:48:52 +00:00

Author

Owner

Copy Link

(b) I have to admit i don't really understand how tree-shaking works, so I will leave it up to you whether or not we should stick to one implementation per exported identifier or not. Just tell me. But I have become fond of the operation name in the object key, rather than in the identifier name. it does look good, and it frees up the identifier name so that we can use globally non-clashing ones and solve the awkwardness of aggregating pocomath-style imports, see #56.

(b) I have to admit i don't really understand how tree-shaking works, so I will leave it up to you whether or not we should stick to one implementation per exported identifier or not. Just tell me. But I have become fond of the operation name in the object key, rather than in the identifier name. it does look good, and it frees up the identifier name so that we can use globally non-clashing ones and solve the awkwardness of aggregating pocomath-style imports, see #56.

glen commented 2022-10-06 12:58:08 +00:00

Author

Owner

Copy Link

(c) I can't really see the virtue of wrapping 95% of implementations in a no-op extra function layer essentially just for documentation of a standard convention of how our implementations are defined. I would much rather, for example, put a comment at the top of every file that uses the convention along the lines of:

/* mathjs implementations are specified by object key-value pairs of the form:
  SIGNATURE: ({DEPENDENCY-SIGNATURES}) => (PARAMETER-LIST) => FUNCTION-BODY
*/

or whatever you think will make the format clear.

(c) I can't really see the virtue of wrapping 95% of implementations in a no-op extra function layer essentially just for documentation of a standard convention of how our implementations are defined. I would much rather, for example, put a comment at the top of _every_ file that uses the convention along the lines of: ``` /* mathjs implementations are specified by object key-value pairs of the form: SIGNATURE: ({DEPENDENCY-SIGNATURES}) => (PARAMETER-LIST) => FUNCTION-BODY */ ``` or whatever you think _will_ make the format clear.

glen commented 2022-10-06 13:20:29 +00:00

Author

Owner

Copy Link

(d) There are two key decisions here that need to be made:

1. Will the revision be written in TypeScript to the fullest extent possible, reverting to plain JavaScript (or TS files with lots of anys and maybe struct null checks off, that sort of thing) only as necessary to deal with the obstacles identified in prototyping?

(If so, I think that's a big enough difference that another round of a "typocomath" prototype is needed before really diving in.)

2. Will mathjs limit itself to only define typed-function types that can precisely correspond to TypeScript types? As current mathjs demonstrates, typed-function has no such limitation inherently, we would only be doing it to make the TypeScript typings of the operations more airtight. if we didn't do it and wanted to have mathjs types like 'identifier' or 'NumInt' then in TypeScript we would need to create so-called "branded" types for these typed-function types and use those in the .d.ts files for the functions that accept these sorts of typed-function types.

Note that (1) and (2) are independent, neither entails the other.

Then if we are going with (1), to avoid redundancy in type-specification, yes we can use typescript compilation with a transformer. it does require using a build command other than raw tsc. But it's only needed for building mathjs, or for folks who want to extend it using the non-redundant syntax, not at all to use mathjs or add a function either written in javascript or with redundant type info.

if we go this way, as far as marking functions that are using typescript type info for signatures, since we are putting signature info in the key, maybe

'add: auto': (a: bigint, b: bigint) => (a+b)

or something along those lines is preferable to

add: inferFromTypescript((a: bigint, b: bigint) => a+b)

? (this suggestion takes a page from modern C++.)

(d) There are two key decisions here that need to be made: 1. Will the revision be written in TypeScript to the fullest extent possible, reverting to plain JavaScript (or TS files with lots of anys and maybe struct null checks off, that sort of thing) _only_ as necessary to deal with the obstacles identified in prototyping? (If so, I think that's a big enough difference that another round of a "typocomath" prototype is needed before really diving in.) 2. Will mathjs limit itself to only define typed-function types that can precisely correspond to TypeScript types? As current mathjs demonstrates, typed-function has no such limitation inherently, we would only be doing it to make the TypeScript typings of the operations more airtight. if we didn't do it and wanted to have mathjs types like 'identifier' or 'NumInt' then in TypeScript we would need to create so-called "branded" types for these typed-function types and use those in the .d.ts files for the functions that accept these sorts of typed-function types. Note that (1) and (2) are independent, neither entails the other. Then if we are going with (1), to avoid redundancy in type-specification, yes we can use typescript compilation with a transformer. it does require using a build command other than raw `tsc`. But it's only needed for building mathjs, or for folks who want to _extend_ it using the non-redundant syntax, not at all to use mathjs or add a function either written in javascript or with redundant type info. if we go this way, as far as marking functions that are using typescript type info for signatures, since we are putting signature info in the key, maybe `'add: auto': (a: bigint, b: bigint) => (a+b)` or something along those lines is preferable to `add: inferFromTypescript((a: bigint, b: bigint) => a+b)` ? (this suggestion takes a page from modern C++.)

glen commented 2022-10-06 13:30:17 +00:00

Author

Owner

Copy Link

(e) To some extent yes this csn be deferred, but as soon as we get started in earnest, we need to decide:

1. What will the actual organization of implementations into files be? I deliberately made pocomath agnostic to this and organized different parts of it different ways. So we need an actual specific organization. My proposal would be src/TYPE/TOPIC.js. So all of the number relational implementations would be in src/number/relational.js, complex arithmetic would be in src/complex/arithmetic.js etc. But I am open to other organizational schemes.

2. where does the documentation of 'compare' (say) go? it has to go somewhere. As much as I dislike separating code and documentation, with the proposal I made in (1) the most logical place might be doc/relational.md...

OK I think that covers the open threads.

(e) To some extent yes this csn be deferred, but as soon as we get started in earnest, we need to decide: 1. What _will_ the actual organization of implementations into files be? I deliberately made pocomath agnostic to this and organized different parts of it different ways. So we need an actual specific organization. My proposal would be `src/TYPE/TOPIC.js`. So all of the number relational implementations would be in `src/number/relational.js`, complex arithmetic would be in `src/complex/arithmetic.js` etc. But I am open to other organizational schemes. 2. where does the documentation of 'compare' (say) go? it has to go somewhere. As much as I dislike separating code and documentation, with the proposal I made in (1) the most logical place might be `doc/relational.md`... OK I think that covers the open threads.

glen commented 2022-10-06 15:07:04 +00:00

Author

Owner

Copy Link

Another comment on (c): if we go with as much TypeScript as possible and auto types via a transformer, there is an alternate syntax for dependencies that removes the redundant listing of operation names and might feel more "self-documenting". Using a non-template subtract just as an example, this would be:

export const fn = {
   'subtract: auto': (dep: {
      add: (a: bigint, b: bigint) => bigint,
      negate: (a: bigint) => bigint}
   }) => (a: bigint, b: bigint) => dep.add(a, dep.negate(b))
}

Note the cost is having to refer to each dependency with the dep. prefix in the function body, but maybe that's a helpful signal of where dependencies are being used, rather than just bloat? I am not saying that I like this better than other alternatives, just continuing to brainstorm to find a format acceptable to everyone.

Another comment on (c): if we go with as much TypeScript as possible and auto types via a transformer, there is an alternate syntax for dependencies that removes the redundant listing of operation names and might feel more "self-documenting". Using a non-template subtract just as an example, this would be: ``` export const fn = { 'subtract: auto': (dep: { add: (a: bigint, b: bigint) => bigint, negate: (a: bigint) => bigint} }) => (a: bigint, b: bigint) => dep.add(a, dep.negate(b)) } ``` Note the cost is having to refer to each dependency with the `dep.` prefix in the function body, but _maybe_ that's a helpful signal of where dependencies are being used, rather than just bloat? I am not saying that I like this better than other alternatives, just continuing to brainstorm to find a format acceptable to everyone.

glen commented 2022-10-06 15:12:58 +00:00

Author

Owner

Copy Link

Note with TypeScript runtime type information, this last proposal is completely distinguishable from a definition with no dependencies like

'negate: auto': (a: number) => -a

and so those would not need a wrapper like direct() either.

Again, if we want a route along these lines, we should do a "typocomath" prototype to make sure we like how it comes out.

Note with TypeScript runtime type information, this last proposal is completely distinguishable from a definition with no dependencies like ``` 'negate: auto': (a: number) => -a ``` and so those would not need a wrapper like `direct()` either. Again, if we want a route along these lines, we should do a "typocomath" prototype to make sure we like how it comes out.

josdejong commented 2022-10-11 09:07:06 +00:00

Copy Link

(f) from a functional point of view dynamic return types based on config are indeed no problem: it works already in pocomath (on a side note... it's impressive that you got all of this working). Main argument is that it would simplify the API (for users) and the and implementation (for ourselves) if we would not support it. That would only be a good idea if we feel it adds little value. Would it be acceptable to you to not support dynamic return types? Or do you feel we really need this?

If we do want to support it, I would love to have support two ways to define the return type: the "simple" and static way is in the signature itself like 'add(bigint, bigint): bigint', and the dynamic way with a Returns callback like it is working now.

(g) I think we do not need an indepth investigation, more like a brief look thoughout the mathjs sourcecode to get a feel on whether how much we can potentially use it.

As for when to migrate: I have the feeling that the migration is already complex enough and that an additional refactoring+reorganization would add more complexities. If that is indeed the case I would prefer doing this refactoring in a second phase. But if it's simpler to directly refactor+reorganize all of the code we could do that right away.

(b) About tree shaking: three shaking can only be applied on the root level constants and functions that are exported. So, if you export three functions from a module, and use only one, you can tree-shake it. But if you export an object holding the three functions you cannot tree-shake it.

// can be tree-shaken
export function add(...) { ... }
export function multiply(...) { ... }
export function divide(...) { ... }

// the functions inside the object cannot be tree-shaken
export const arithmeticFns = {
  add: ...,
  multiply: ...,
  divide: ...
}

You have a good point about naming collisions with imports. I have a feeling too that grouping functions in objects will work nicely. As long as we group them in meaningful groups of functions that are typically used together that will work just fine: in that case there is no need to treeshake individual functions from these objects, so nothing to worry about in that regard.

(d.1) In my opion it would be good to use TypeScript in mathjs in a pragmatic way. Adding types to functions and everything helps a lot in communicating what the function expects, and it prevents a specific category of bugs. It is helpful in bigger teams and projects. I myself have a bit of a love/hate relation with TypeScript because of the language not being sound, and the JS+TS ecosystem being a hybrid where you regularly have to resort to @ts-ignore, using any, and add ugly type casting just for the sake of making the TS compiler happy. So I would say, let's use it where it helps, and be pragmatic where it hurts. What is your opinion Glen?

(d.2) I think it is essential that using the libary or extending the libary does not require using a special TS transformer. At it's core, typed-function and pocomath should be able to work without a transformer and without TS, and something like inferFromTypescript should work on top of that. In mathjs it will be very valuable if we can make this work and I'm totally fine with configuring and using some transformer there! It sounds like the best of both worlds.

I do love your proposed syntax like add: inferFromTypescript((a: bigint, b: bigint) => a+b) 😎. The code itself makes very clear what's happening, and if you try to use it without the transformer it will probably be possible to throw an informative error explaining what you should do to make it work.

(e.1) I agree with your proposal:

My proposal would be src/TYPE/TOPIC.js. So all of the number relational implementations would be in src/number/relational.js,

And additionally:

• we will have a category of generic functions not bound to a single data type, that can go in src/generic/TOPIC.js
• when a file TOPIC.js grows too large, we can split it in smaller files (probably per function, like src/TYPE/TOPIC/FUNCTION.js

(e.2) Since there will not be a single implementation of a function but multiple, one per data type in many cases, I think we'll have to conclude that we have to separate code and documentation. I think we can hold on to a stucture similar to the current structure of the docs, except that we do not autogenerate the contents in docs/reference/functions but actually write the docs there. And instead, we can probably autogenerate the embedded docs. Sometimes, the examples in the embedded docs differ from the code examples, because of the function transforms. We'll need to have a way to override those examples with examples using the transformed functions.

(c.1) yes makes sense. Ok then lets keep the dependency callback as it is right now: as an anonymous function, always required.

(c.2) About your last two proposals for a TS based syntax: I like that. I think the (dep: ...) notation is fine. It will be much better than a lot of duplication. I expect there will be some limitations in being able to map TS types to typed-function types. It will be good indeed to work out a pocomath experiment to see if we can indeed neatly integrate a TS transformer with typed-function+pocomath. To me that feels like the right way ahead. Is this something you would like to give a try?

**(f)** from a functional point of view dynamic return types based on config are indeed no problem: it works already in pocomath (on a side note... it's impressive that you got all of this working). Main argument is that it would simplify the API (for users) and the and implementation (for ourselves) if we would not support it. That would only be a good idea if we feel it adds little value. Would it be acceptable to you to not support dynamic return types? Or do you feel we _really_ need this? If we do want to support it, I would love to have support two ways to define the return type: the "simple" and static way is in the signature itself like `'add(bigint, bigint): bigint'`, and the dynamic way with a `Returns` callback like it is working now. **(g)** I think we do not need an indepth investigation, more like a brief look thoughout the mathjs sourcecode to get a feel on whether how much we can _potentially_ use it. As for when to migrate: I have the feeling that the migration is already complex enough and that an additional refactoring+reorganization would add more complexities. If that is indeed the case I would prefer doing this refactoring in a second phase. But if it's simpler to directly refactor+reorganize all of the code we could do that right away. **(b)** About tree shaking: three shaking can only be applied on the root level constants and functions that are exported. So, if you export three functions from a module, and use only one, you can tree-shake it. But if you export an object holding the three functions you cannot tree-shake it. ```js // can be tree-shaken export function add(...) { ... } export function multiply(...) { ... } export function divide(...) { ... } // the functions inside the object cannot be tree-shaken export const arithmeticFns = { add: ..., multiply: ..., divide: ... } ``` You have a good point about naming collisions with imports. I have a feeling too that grouping functions in objects will work nicely. As long as we group them in meaningful groups of functions that are typically used together that will work just fine: in that case there is no need to treeshake individual functions from these objects, so nothing to worry about in that regard. **(d.1)** In my opion it would be good to use TypeScript in mathjs in a pragmatic way. Adding types to functions and everything helps a lot in communicating what the function expects, and it prevents a specific category of bugs. It is helpful in bigger teams and projects. I myself have a bit of a love/hate relation with TypeScript because of the language not being sound, and the JS+TS ecosystem being a hybrid where you regularly have to resort to `@ts-ignore`, using `any`, and add ugly type casting just for the sake of making the TS compiler happy. So I would say, let's use it where it helps, and be pragmatic where it hurts. What is your opinion Glen? **(d.2)** I think it is essential that using the libary or extending the libary does not _require_ using a special TS transformer. At it's core, typed-function and pocomath should be able to work without a transformer and without TS, and something like `inferFromTypescript` should work _on top of that_. In mathjs it will be very valuable if we can make this work and I'm totally fine with configuring and using some transformer there! It sounds like the best of both worlds. I do love your proposed syntax like `add: inferFromTypescript((a: bigint, b: bigint) => a+b)` :sunglasses:. The code itself makes very clear what's happening, and if you try to use it without the transformer it will probably be possible to throw an informative error explaining what you should do to make it work. **(e.1)** I agree with your proposal: > My proposal would be `src/TYPE/TOPIC.js`. So all of the number relational implementations would be in `src/number/relational.js`, And additionally: - we will have a category of generic functions not bound to a single data type, that can go in `src/generic/TOPIC.js` - when a file `TOPIC.js` grows too large, we can split it in smaller files (probably per function, like `src/TYPE/TOPIC/FUNCTION.js` **(e.2)** Since there will not be a single implementation of a function but multiple, one per data type in many cases, I think we'll have to conclude that we have to separate code and documentation. I think we can hold on to a stucture similar to the current structure of [the docs](https://github.com/josdejong/mathjs/tree/develop/docs), except that we do not autogenerate the contents in `docs/reference/functions` but actually write the docs there. And instead, we can probably autogenerate [the embedded docs](https://github.com/josdejong/mathjs/tree/develop/src/expression/embeddedDocs). Sometimes, the examples in the embedded docs differ from the code examples, because of the function transforms. We'll need to have a way to override those examples with examples using the transformed functions. **(c.1)** yes makes sense. Ok then lets keep the dependency callback as it is right now: as an anonymous function, always required. **(c.2)** About your last two proposals for a TS based syntax: I like that. I think the `(dep: ...)` notation is fine. It will be _much_ better than a lot of duplication. I expect there will be some limitations in being able to map TS types to typed-function types. It will be good indeed to work out a pocomath experiment to see if we can indeed neatly integrate a TS transformer with typed-function+pocomath. To me that feels like the right way ahead. Is this something you would like to give a try?

glen commented 2022-10-11 12:49:17 +00:00

Author

Owner

Copy Link

Sure i am happy to do a prototype implementing the syntax from #55 (comment) and #55 (comment) if you like that. But I am unclear where you've landed as far as whether mathjs types should be restricted to correspond precisely to TypeScript types. To put it concretely, in the new typocomath prototype, should I try to include the NumInt type (a number which happens to represent an integer, which can be defined via a test, so is fine in current typed-function, but is impossible to capture precisely in TypeScript: you can make the type of literal integer-numbers with template literal types, and you can make "branded types" that state what they are, but you can't make full-fledged types that enforce the restriction)? Or should I discard the NumInt type since it won't ever precisely correpond to a TypeScript type?

Sure i am happy to do a prototype implementing the syntax from https://code.studioinfinity.org/glen/pocomath/issues/55#issuecomment-799 and https://code.studioinfinity.org/glen/pocomath/issues/55#issuecomment-800 if you like that. But I am unclear where you've landed as far as whether mathjs types should be restricted to correspond precisely to TypeScript types. To put it concretely, in the new typocomath prototype, should I try to include the NumInt type (a number which happens to represent an integer, which can be defined via a test, so is fine in current typed-function, but is impossible to capture precisely in TypeScript: you can make the type of **literal** integer-numbers with template literal types, and you can make "branded types" that **state** what they are, but you can't make full-fledged types that enforce the restriction)? Or should I discard the NumInt type since it won't ever precisely correpond to a TypeScript type?

glen commented 2022-10-11 13:01:25 +00:00

Author

Owner

Copy Link

On use of ts and transformers amd such: if we write the bulk of implementations in typescript, then naturally people will write extensions in typescript. but yes we can presumably engineer things so that the automatic discovery of signature types works only with the transformer but you could write all signatures out (redundantly) and get an extension working without the transformer, with a message that auto signatures only work with the transformer if you try to use one without.

but to be clear, even with transformers, we are adopting a plan here where mathjs will not "Just Work" in typescript: to get the resulting mathjs to be callable from typescript, you will be obliged to do a build step that critically uses plain JavaScript to write out an auto-generated .d.ts file that will have the declarations needed to call the corresponding mathjs object from typescript. I think we are clear and in agreement on that, just want to make sure.

On use of ts and transformers amd such: if we write the bulk of implementations in typescript, then naturally people will write extensions in typescript. but yes we can presumably engineer things so that the automatic discovery of signature types works only with the transformer but you could write all signatures out (redundantly) and get an extension working without the transformer, with a message that auto signatures only work with the transformer if you try to use one without. but to be clear, even with transformers, we are adopting a plan here where mathjs will **not** "Just Work" in typescript: to get the resulting mathjs to be callable from typescript, you will be obliged to do a build step that critically uses plain JavaScript to write out an auto-generated .d.ts file that will have the declarations needed to call the corresponding mathjs object from typescript. I think we are clear and in agreement on that, just want to make sure.

josdejong commented 2022-10-13 12:03:53 +00:00

Copy Link

I am unclear where you've landed as far as whether mathjs types should be restricted to correspond precisely to TypeScript types.

At this moment I do not have a clear overview of where possible difficulties arise in this regard. In general, I think we should be pragmatic: try to recon with TypeScript but let it not dictate the direction we want to take too much. Staying aligned as much as possible with plain JavaScript and Typescript types will help though, so I guess we have to discuss individual cases.

should I discard the NumInt type since it won't ever precisely correpond to a TypeScript type?

I would say: no. I think it is perfectly fine to introduce a NumInt type if that is useful for mathjs. I think I'm still not understanding something here: I would think with a typeguard you can enforce any restrictions?

but to be clear, even with transformers, we are adopting a plan here where mathjs will not "Just Work" in typescript: to get the resulting mathjs to be callable from typescript, you will be obliged to do a build step that critically uses plain JavaScript to write out an auto-generated. d.ts file that will have the declarations needed to call the corresponding mathjs object from typescript. I think we are clear and in agreement on that, just want to make sure.

Good to double check. Indeed, mathjs will not "Just Work" in typescript, because we concluded that we cannot make typed-function+pocomath work with TS in a statically defined way (right?). This sounds scary but I think there is a bit more to this:

1. you can use mathjs in TypeScript straight ahead: all functions it offers come with full-fledged type definitions (generated via a build script)
2. you can define a new typed-function in any JS or TS environment without need for a special build steps or adapters but this has limitations:
   • creating a new typed-function with typed(...) will only return a generic type defintion like function without any details. To create a full-fledged type definition, you'll have to run a build script, which instantiates the typed-function, calls a method like .getTypeDefinitions() on the function, and writes the output to a .d.ts file. I think mathjs should provide some utilities to streamline this as much as possible to lower the barrier, I'm not exactly sure yet how that will pan out though.
   • By default, in both a JS and TS environment, you have to define signatures as strings. Automatically inferring the types from your TypeScript code using inferFromTypescript(...) requires configuring a TS transformer.

Do you agree with this approach Glen, or do you feel like we're on the wrong road?

> I am unclear where you've landed as far as whether mathjs types should be restricted to correspond precisely to TypeScript types. At this moment I do not have a clear overview of where possible difficulties arise in this regard. In general, I think we should be pragmatic: try to recon with TypeScript but let it not dictate the direction we want to take too much. Staying aligned as much as possible with plain JavaScript and Typescript types will help though, so I guess we have to discuss individual cases. > should I discard the NumInt type since it won't ever precisely correpond to a TypeScript type? I would say: no. I think it is perfectly fine to introduce a `NumInt` type if that is useful for mathjs. I think I'm still not understanding something here: I would think with a typeguard you can enforce any restrictions? > but to be clear, even with transformers, we are adopting a plan here where mathjs will not "Just Work" in typescript: to get the resulting mathjs to be callable from typescript, you will be obliged to do a build step that critically uses plain JavaScript to write out an auto-generated. d.ts file that will have the declarations needed to call the corresponding mathjs object from typescript. I think we are clear and in agreement on that, just want to make sure. Good to double check. Indeed, _mathjs will not "Just Work" in typescript_, because we concluded that we cannot make typed-function+pocomath work with TS in a statically defined way (right?). This sounds scary but I think there is a bit more to this: 1. you _can_ use mathjs in TypeScript straight ahead: all functions it offers come with full-fledged type definitions (generated via a build script) 2. you _can_ define a new typed-function in any JS or TS environment without need for a special build steps or adapters but this has limitations: - creating a new typed-function with `typed(...)` will only return a generic type defintion like `function` without any details. To create a full-fledged type definition, you'll have to run a build script, which instantiates the typed-function, calls a method like `.getTypeDefinitions()` on the function, and writes the output to a `.d.ts` file. I think mathjs should provide some utilities to streamline this as much as possible to lower the barrier, I'm not exactly sure yet how that will pan out though. - By default, in both a JS and TS environment, you have to define signatures as strings. Automatically inferring the types from your TypeScript code using `inferFromTypescript(...)` requires configuring a TS transformer. Do you agree with this approach Glen, or do you feel like we're on the wrong road?

glen commented 2022-10-13 13:40:40 +00:00

Author

Owner

Copy Link

ok, i will make the typocomath prototype have a NumInt type (even though it is no such type in mathjs right now) so we can see how non-TypeScriptable types go.

As to your question about the limitation here, typed-functions will be able to dispatch on a NumInt, but if you have a typed-function like say gcd with only a NumInt implementation and no number implementation, resulting in a TypeScript declaration that only allows NumInt arguments, then the compiler will not let you write gcd(3+9, 4+4) because it has no way of knowing that a sum is a NumInt because the typescript definition of NumInt will just be "a thing that says it is a NumInt", not any actual TypeScript type (since there is no such type). You would have to write gcd(3+9 as NumInt, 4+4 as NumInt) or put those values in variables declared as NumInts or something along those lines to get the TypeScript compiler to let you make the call (even though the JavaScript will handle it perfectly well once you manage to get into the function).

And as for your summary under "Good to double check", I agree with everything but I thought we had landed on using the signature ": auto" to signal inferring from typescript that only works with the transformer configured? that was the last post in that thread... anyhow I will implement it that way in the prototype so you can see how you like it, we could always switch to another notation.

so i think i have everything needed to start on the prototype. it will go in stages and take a little while, of course. should i do the benchmark on the pocomath prototype first, to be sure that hasn't already introduced some big unknown inefficiency?

ok, i will make the typocomath prototype have a NumInt type (even though it is no such type in mathjs right now) so we can see how non-TypeScriptable types go. As to your question about the limitation here, typed-functions will be able to dispatch on a NumInt, but if you have a typed-function like say gcd with only a NumInt implementation and no number implementation, resulting in a TypeScript declaration that only allows NumInt arguments, then the compiler will not let you write `gcd(3+9, 4+4)` because it has no way of knowing that a sum is a NumInt because the typescript definition of NumInt will just be "a thing that says it is a NumInt", not any actual TypeScript type (since there is no such type). You would have to write `gcd(3+9 as NumInt, 4+4 as NumInt)` or put those values in variables declared as NumInts or something along those lines to get the TypeScript compiler to let you make the call (even though the JavaScript will handle it perfectly well once you manage to get into the function). And as for your summary under "Good to double check", I agree with everything but I thought we had landed on using the signature ": auto" to signal inferring from typescript that only works with the transformer configured? that was the last post in that thread... anyhow I will implement it that way in the prototype so you can see how you like it, we could always switch to another notation. so i think i have everything needed to start on the prototype. it will go in stages and take a little while, of course. should i do the benchmark on the pocomath prototype first, to be sure that hasn't already introduced some big unknown inefficiency?

josdejong commented 2022-10-13 15:02:35 +00:00

Copy Link

About NumInt: sounds good to try it out in the prototype to see how it works out, thanks.

Thanks for your explanation, I think I understand what you mean. To me, having something like NumInt is not different from having say BigNumber. If we have a gcd function that only works with BigNumbers, gcd(12, 8) will not work even though we know can safely convert those integer numbers into a BigNumber. We have to write gcd(bignumber('12'), bignumber('8')). I think in the NumInt variant, we should create a class for it and use it like gcd(NumInt(12), NumInt(8)). So, introducing a type like NumInt over using number comes at a cost. But it can be worth it if the function that requires this data type then has certain mathematical or performance advantages.

About : auto: I like the more explicit inferFromTypescript(...) better myself, but indeed this is "just" syntax and not essential to have decided upon upfront. I may change my mind if I see how : auto works out in practice thoguh, who knows 😉

About the benchmark: yes makes sense to first do the benchmark. If it turns out that there is a huge (unexpected) performance problem, we may be forced to rethink the architecture. I don't expect that but you never know.

**About NumInt:** sounds good to try it out in the prototype to see how it works out, thanks. Thanks for your explanation, I think I understand what you mean. To me, having something like `NumInt` is not different from having say `BigNumber`. If we have a `gcd` function that only works with BigNumbers, `gcd(12, 8)` will not work even though we know can safely convert those integer numbers into a `BigNumber`. We have to write `gcd(bignumber('12'), bignumber('8'))`. I think in the `NumInt` variant, we should create a class for it and use it like `gcd(NumInt(12), NumInt(8))`. So, introducing a type like `NumInt` over using `number` comes at a cost. But it can be worth it if the function that requires this data type then has certain mathematical or performance advantages. **About `: auto`:** I like the more explicit `inferFromTypescript(...)` better myself, but indeed this is "just" syntax and not essential to have decided upon upfront. I may change my mind if I see how `: auto` works out in practice thoguh, who knows :wink: **About the benchmark:** yes makes sense to first do the benchmark. If it turns out that there is a huge (unexpected) performance problem, we may be forced to rethink the architecture. I don't expect that but you never know.

glen commented 2022-10-13 19:17:52 +00:00

Author

Owner

Copy Link

Ok, I think we are seeing eye-to-eye on something like NumInt. I will use that as the example in the typocomath prototype because I already used it in pocomath, so that will make the new prototype a bit easier to go through, even if we don't actually end up with precisely a NumInt type in the future mathjs. (I picked gcd as the example here because basically gcd isn't really defined on all numbers; it's really only defined on integers, or on exact rational numbers, e.g. the Fraction type.)

And thanks for giving "auto" a look when I get it working, we can always switch notations.

And OK I will do the benchmark first; that makes sense. The current plan of benchmarking polynomial root-finding will involve a PR to mathjs, as it does not have a root-finding function so far as I can tell. I'll try to get that done as soon as I can but I am coming up on a book deadline, so my progress will be slowed. (I will try to keep it more steady, just slower.)

Any case, seems like my direction for the next few weeks is clear, and by then we should have feedback from m93a if we are going to get any, so I am feeling good about the plans.

Ok, I think we are seeing eye-to-eye on something like NumInt. I will use that as the example in the typocomath prototype because I already used it in pocomath, so that will make the new prototype a bit easier to go through, even if we don't actually end up with precisely a NumInt type in the future mathjs. (I picked gcd as the example here because basically gcd isn't really defined on all numbers; it's really only defined on integers, or on exact rational numbers, e.g. the Fraction type.) And thanks for giving "auto" a look when I get it working, we can always switch notations. And OK I will do the benchmark first; that makes sense. The current plan of benchmarking polynomial root-finding will involve a PR to mathjs, as it does not have a root-finding function so far as I can tell. I'll try to get that done as soon as I can but I am coming up on a book deadline, so my progress will be slowed. (I will try to keep it more steady, just slower.) Any case, seems like my direction for the next few weeks is clear, and by then we should have feedback from m93a if we are going to get any, so I am feeling good about the plans.

josdejong commented 2022-10-18 10:01:17 +00:00

Copy Link

Thanks a lot, sounds good. Good luck with finishing your book!

Thanks a lot, sounds good. Good luck with finishing your book!

glen commented 2022-12-02 02:04:06 +00:00

Author

Owner

Copy Link

The results are in:

mathjs (current main):

mathjs> node test/benchmark/roots.js
There are 3197 roots of the 1080 integer cubic
polynomials (with coefficients <= 5 )
count roots                              x 36.74 ops/sec ±0.18% (65 runs sampled)

pocomath:

pocomath> node benchmark/roots.mjs
There are 3197 roots of the 1080 integer cubic
polynomials (with coefficients <= 5 )
count roots                              x 348 ops/sec ±0.30% (94 runs sampled)

A factor of 9 speedup! That's gotta be at least partially "real" (i.e., not just because pocomath is a "scaled-down mockup"). The top-level code in polynomialRoot is essentially identical (already checked in to pocomath, you can look) and the test suite is as well (I'll check it in momentarily). So there's nothing particularly done to speed things up in pocomath. Thus I assume the speedup is almost entirely because of typing the implementations, so that when dependencies are resolved they grab the specific implementations for the types they need, avoiding typed-function resolution for all calls further down the chain.

(Aside: this outcome really really really makes me wonder why the TypeScript folks are so absolutely against runtime type dispatch. It's a decision that makes writing TypeScript code more cumbersome, and they are clearly leaving performance gains to be had on the table. Ah well, their loss, but it completely confirms that I will never choose TypeScript for a project where I have influence on the language chosen. End of aside.)

So things look good. I will kick off implementation of "typocomath" per all of our discussions as soon as I can. I hope that I can assemble all of the pieces that we wanted in the final round of prototyping; sorry it's been a while. But I think we kept pretty good notes in our discussions. The key is that here you do want just about all of the code of the final prototype to be TypeScript, with just escapes into JavaScript when necessary for the sake of generating the typed-functions.

One general big-picture question: What do you think the scope of the "typocomath" prototype should be? Roughly the same as this final scope of "pocomath", i.e. number, NumInt, bigint, Fraction, Complex, some kind of collection, subtypes, generics, generics with bounded type parameters, return type annotations, allow plain function implementations if desired, Chains, and enough operations to again run the "roots" benchmark? Or more than that or less than that?

And some specific detailed questions:

1. In Pocomath, the collection I implemented was Tuple, a 1-dimensional type-homogeneous array. This doesn't have an exact analogue in mathjs, as Arrays can be type-inhomogeneous and Matrix can be multidimensional and is optionally type-homogeneous or type-inhomogeneous. I assume we will want to stick with those two collection types in mathjs 12 (or whatever we will call the big refactor)? If so, in the new prototype should I try to implement just a type-dumb Array, or just a Matrix-lite that can be type-homogeneous or not (for the prototype I would probably call it Vector and keep it 1d, so that the project doesn't get too huge). Or should I just stick with Tuple for the prototype even though we presumably won't be using it in the big refactor?

2. In Pocomath, generics are restricted to have exactly one type parameter, that must be called T. I suspect 99% or even 100% of mathjs could be reasonably typed with this restriction, although I can easily imagine people finding the restriction artificial/ugly/limiting/etc. Should I make Typocomath with this restriction, or allow arbitrarily many type parameters in the generics? (I will be honest that I am a bit afraid that allowing mutiple type parameters will noticeably increase the coding time without a great deal of practical advantage. On the other hand, if you feel that the actual release of mathjsNext should definitely allow multiple type parameters, I should probably implement it in the prototype so we get whatever hitches that brings worked out in advance.)

3. Since I was just playing and writing from scratch, I experimented with stuff like making subtraction and division generic (adding/multiplying by negation/inverse). That's not really how to do it for practical reasons (like JavaScript offering builtin operations for these). Should I hew more closely to the mathjs implementation details of specific operations in the Typocomath prototype? Or just go ahead and implement things however is easiest as long as they work? (Will we explicitly try to keep the same internal algorithms in the big refactor, or change them if/as convenient as long as the unit tests pass?)

4. Finally, in Pocomath I deliberately organized some of the items by type with one operation per file in each type, some of the items by operation with all the types in one file, some in big groups of functions, etc. That's of course crazy, we'd never really mix organizational schemes like this. So in this prototype I was thinking that I should just organize it the way we would actually like mathjs to be organized after the refactor, since we now have a completely free choice of that organizational scheme, thanks to the flexibility of the composition mechanism here. So what organizational scheme do we want? I think we discussed this a little: top-level division by type, with the operations for a type divided into several files, one for each major group of functions, but not so finely as one file per function? If that wasn't what we discussed or on reflection isn't your preferred organization, I am really quite open to whatever chopping up of all of the implementations into files you like. Please just let me know. (And the latest exercise made me realize we need to decide where cross-type operations go, like arg() that takes a Complex<T> and returns a real number (of some type; it can't be T because it makes no sense for the arg of a Gaussian integer Complex<bigint> to be a bigint) or cis that takes a real number and returns a complex number, etc. So if you have any thoughts on that point, let me know.

Thanks and looking forward to moving ahead with this!

The results are in: mathjs (current main): ``` mathjs> node test/benchmark/roots.js There are 3197 roots of the 1080 integer cubic polynomials (with coefficients <= 5 ) count roots x 36.74 ops/sec ±0.18% (65 runs sampled) ``` pocomath: ``` pocomath> node benchmark/roots.mjs There are 3197 roots of the 1080 integer cubic polynomials (with coefficients <= 5 ) count roots x 348 ops/sec ±0.30% (94 runs sampled) ``` A factor of 9 speedup! That's gotta be at least partially "real" (i.e., not just because pocomath is a "scaled-down mockup"). The top-level code in polynomialRoot is essentially _identical_ (already checked in to pocomath, you can look) and the test suite is as well (I'll check it in momentarily). So there's nothing _particularly_ done to speed things up in pocomath. Thus I assume the speedup is almost entirely because of typing the implementations, so that when dependencies are resolved they grab the specific implementations for the types they need, avoiding typed-function resolution for all calls further down the chain. (Aside: this outcome really _really_ **really** makes me wonder why the TypeScript folks are so absolutely against runtime type dispatch. It's a decision that makes writing TypeScript code more cumbersome, and they are clearly leaving performance gains to be had on the table. Ah well, their loss, but it completely confirms that I will never choose TypeScript for a project where I have influence on the language chosen. End of aside.) So things look good. I will kick off implementation of "typocomath" per all of our discussions as soon as I can. I hope that I can assemble all of the pieces that we wanted in the final round of prototyping; sorry it's been a while. But I think we kept pretty good notes in our discussions. The key is that here you _do_ want just about all of the code of the final prototype to be TypeScript, with just escapes into JavaScript when necessary for the sake of generating the typed-functions. One general big-picture question: What do you think the scope of the "typocomath" prototype should be? Roughly the same as this final scope of "pocomath", i.e. number, NumInt, bigint, Fraction, Complex<T>, some kind of collection, subtypes, generics, generics with bounded type parameters, return type annotations, allow plain function implementations if desired, Chains, and enough operations to again run the "roots" benchmark? Or more than that or less than that? And some specific detailed questions: 1) In Pocomath, the collection I implemented was Tuple<T>, a 1-dimensional _type-homogeneous_ array. This doesn't have an exact analogue in mathjs, as Arrays can be type-inhomogeneous and Matrix can be multidimensional and is optionally type-homogeneous or type-inhomogeneous. I assume we will want to stick with those two collection types in mathjs 12 (or whatever we will call the big refactor)? If so, in the new prototype should I try to implement just a type-dumb Array, or just a Matrix-lite that can be type-homogeneous or not (for the prototype I would probably call it Vector and keep it 1d, so that the project doesn't get too huge). Or should I just stick with Tuple for the prototype even though we presumably won't be using it in the big refactor? 2) In Pocomath, generics are restricted to have exactly one type parameter, that must be called `T`. I suspect 99% or even 100% of mathjs could be reasonably typed with this restriction, although I can easily imagine people finding the restriction artificial/ugly/limiting/etc. Should I make Typocomath with this restriction, or allow arbitrarily many type parameters in the generics? (I will be honest that I am a bit afraid that allowing mutiple type parameters will noticeably increase the coding time without a great deal of practical advantage. On the other hand, if you feel that the actual release of mathjsNext should definitely allow multiple type parameters, I should probably implement it in the prototype so we get whatever hitches that brings worked out in advance.) 3) Since I was just playing and writing from scratch, I experimented with stuff like making subtraction and division generic (adding/multiplying by negation/inverse). That's not really how to do it for practical reasons (like JavaScript offering builtin operations for these). Should I hew more closely to the mathjs implementation details of specific operations in the Typocomath prototype? Or just go ahead and implement things however is easiest as long as they work? (Will we explicitly try to keep the same internal algorithms in the big refactor, or change them if/as convenient as long as the unit tests pass?) 4) Finally, in Pocomath I deliberately organized some of the items by type with one operation per file in each type, some of the items by operation with all the types in one file, some in big groups of functions, etc. That's of course crazy, we'd never really mix organizational schemes like this. So in this prototype I was thinking that I should just organize it the way we would actually like mathjs to be organized after the refactor, since we now have a completely free choice of that organizational scheme, thanks to the flexibility of the composition mechanism here. So what organizational scheme do we want? I think we discussed this a little: top-level division by type, with the operations for a type divided into several files, one for each major group of functions, but not so finely as one file per function? If that wasn't what we discussed or on reflection isn't your preferred organization, I am really quite open to whatever chopping up of all of the implementations into files you like. Please just let me know. (And the latest exercise made me realize we need to decide where cross-type operations go, like arg() that takes a `Complex<T>` and returns a real number (of some type; it can't be `T` because it makes no sense for the arg of a Gaussian integer `Complex<bigint>` to be a bigint) or `cis` that takes a real number and returns a complex number, etc. So if you have any thoughts on that point, let me know. Thanks and looking forward to moving ahead with this!

glen commented 2022-12-02 02:22:53 +00:00

Author

Owner

Copy Link

Oh, I think we actually had above earlier a fair amount of discussion of point 4 in my latest note. It seems we settled on src/TYPE/TOPIC.ts, so that's fine. But all of the other questions stand, and in addition, if you have any thoughts about where to put things like arg and cis that map between types (these I assume should go in complex because they mostly come up in the context of complex numbers, unless we want some kind of stricter rule of where functions go based on their argument and return types...).

Oh, I think we actually had above earlier a fair amount of discussion of point 4 in my latest note. It seems we settled on src/TYPE/TOPIC.ts, so that's fine. But all of the other questions stand, and in addition, if you have any thoughts about where to put things like arg and cis that map between types (these I assume should go in complex because they mostly come up in the context of complex numbers, unless we want some kind of stricter rule of where functions go based on their argument and return types...).

josdejong commented 2022-12-02 09:43:58 +00:00

Copy Link

whoooow, that is unbelievable! Man, that refactoring will be totally worth it! Not only a much better architecture, but also better performance 😄 Please get yourself a beer on my cost. I suppose we may lose a bit of performance when fully working out all data types in all functions, but that will probably be not much (if any).

About your "Aside": yes, that is really interesting knowledge. Makes me wonder if we can somehow develop a generic "TypeScript runtime type dispatch" library, instead of a mathjs-taylored solution. On the other hand, we do have a quite specific use case and the issue is probably not very main stream 😅 .

It's been a while for me too, the plan for typocomath. I think we can summarize it as follows:

1. We go for a hybrid JS/TS approach
2. In the basis, the typocomath core has a pure JS API with objects { signature: fnImpl }. On top of that, it has a method inferFromTypescript() or notation like auto to automatically infer the types of a signature from TypeScript
3. When creating a typed-function dynamically, it does not have a rich type definition, but just something like function.
4. A typed-function has a method like .getTypeDefinitions(), which outputs a string with rich TypeScript definitions. With a build script we can generate rich types for all build-in functions and write that into files for the npm package.

For reference, here the links to the relevant discussions:

• #55 (this one)
• https://github.com/josdejong/mathjs/discussions/2741
• https://github.com/josdejong/mathjs/discussions/2742
• https://github.com/josdejong/typed-function/issues/123#issuecomment-1268146712
• https://github.com/josdejong/typed-function/issues/124
• glen/typocomath#1

Scope for the prototype:
First: still have to say I love the name again typocomath 😂 . I would say this prototype can be a less extensive prototype than pocomath, and should aim at validating that the hybrid TypeScript/JS/typed-function approach works. Right after the prototype we want to move it into mathjs for real, so any work on the prototype that can be reused in mathjs in the end is not wasted effort.

Question 1: I would love to rethink the matrix classes and do things differently with the knowledge we have now. But if we change the structure of matrices, it will touch a large part of the code base. I think to keep this refactor limited it would be good to keep the matrices as is for now, there are a lot of moving parts already. There is a mechanism in Matrix right now to keep track on whether the matrix contains homogenious types, it would be nice if that can be reflected in the types if possible. What do you think?

Question 2: I would love to start simple, with just T. And see if we want to extend this some day in the future when there is a concrete need for it.

Question 3: Ha ha, yes I noticed. As a rule of thumb: for the refactor I think it will be helpful to keep the functions themselves as-is as much as possible, so we don't touch the logic of the functions but only of the architecture. But if you see some low hanging fruit to improve something with little effort, that's totally fine with me. We just need to be careful not to make the refactor too big to handle.

Question 4: This is a nice opportunity to reorganize the code, though it will be just as easy afterwards to reorganize something :). Here I'm a bit in a split: I would like to minimize the refactor, but also work towards the structure that we want in the end. The rough "ideal" structure that I have in mind is: group the basic arithmetic functions per data type (add, subtract, multiply, (also trigo?), etc), and group the generic functions per category (algebra, relational, statistics, set, ...). I'm not entirly sure where cross-type functions should go, I think maybe best is to handle them like a generic function, grouped "per category"?

whoooow, that is unbelievable! Man, that refactoring will be totally worth it! Not only a much better architecture, but also better performance 😄 Please get yourself a beer on my cost. I suppose we may lose a bit of performance when fully working out all data types in all functions, but that will probably be not much (if any). About your "Aside": yes, that is really interesting knowledge. Makes me wonder if we can somehow develop a generic "TypeScript runtime type dispatch" library, instead of a mathjs-taylored solution. On the other hand, we do have a quite specific use case and the issue is probably not very main stream 😅 . It's been a while for me too, the plan for typocomath. I think we can summarize it as follows: 1. We go for a hybrid JS/TS approach 2. In the basis, the `typocomath` core has a pure JS API with objects `{ signature: fnImpl }`. On top of that, it has a method `inferFromTypescript()` or notation like `auto` to automatically infer the types of a signature from TypeScript 3. When creating a typed-function dynamically, it does not have a rich type definition, but just something like `function`. 4. A typed-function has a method like `.getTypeDefinitions()`, which outputs a string with rich TypeScript definitions. With a build script we can generate rich types for all build-in functions and write that into files for the npm package. For reference, here the links to the relevant discussions: - https://code.studioinfinity.org/glen/pocomath/issues/55 (this one) - https://github.com/josdejong/mathjs/discussions/2741 - https://github.com/josdejong/mathjs/discussions/2742 - https://github.com/josdejong/typed-function/issues/123#issuecomment-1268146712 - https://github.com/josdejong/typed-function/issues/124 - https://code.studioinfinity.org/glen/typocomath/issues/1 Scope for the prototype: First: still have to say I love the name again `typocomath` 😂 . I would say this prototype can be a less extensive prototype than pocomath, and should aim at validating that the hybrid TypeScript/JS/typed-function approach works. Right after the prototype we want to move it into mathjs for real, so any work on the prototype that can be reused in mathjs in the end is not wasted effort. Question 1: I would _love_ to rethink the matrix classes and do things differently with the knowledge we have now. But if we change the structure of matrices, it will touch a large part of the code base. I think to keep this refactor limited it would be good to keep the matrices as is for now, there are a lot of moving parts already. There is a mechanism in Matrix right now to keep track on whether the matrix contains homogenious types, it would be nice if that can be reflected in the types if possible. What do you think? Question 2: I would love to start simple, with just `T`. And see if we want to extend this some day in the future when there is a concrete need for it. Question 3: Ha ha, yes I noticed. As a rule of thumb: for the refactor I think it will be helpful to keep the functions themselves as-is as much as possible, so we don't touch the logic of the functions but only of the architecture. But if you see some low hanging fruit to improve something with little effort, that's totally fine with me. We just need to be careful not to make the refactor too big to handle. Question 4: This is a nice opportunity to reorganize the code, though it will be just as easy afterwards to reorganize something :). Here I'm a bit in a split: I would like to minimize the refactor, but also work towards the structure that we want in the end. The rough "ideal" structure that I have in mind is: group the basic arithmetic functions per data type (add, subtract, multiply, (also trigo?), etc), and group the generic functions per category (algebra, relational, statistics, set, ...). I'm not entirly sure where cross-type functions should go, I think maybe best is to handle them like a generic function, grouped "per category"?

glen commented 2022-12-02 13:42:55 +00:00

Author

Owner

Copy Link

OK, perfect. I will go for making typocomath somewhat less extensive, driving toward the polynomial solver to replicate the benchmark, and try to include a matrix-lite class called Vector which can be either homogeneous or inhomogeneous. (The plan there is to make it a template Vector<T> but allow T to be any to cover the inhomogeneous case; in pocomath, substitution of any for the type parameter was specifically disallowed. I will try for it still to be disallowed by default, since I still don't think Complex<Any> makes sense, but with an escape mechanism to specifically permit it for a specific template like Vector.)

I won't try to grab any of the implementation of Matrix, but for the rest I will try to stick to the mathjs implementations except if as you say there seems to be something really worthwhile that the architecture allows.

Finally, I will try as a first pass to organize the implementations of the operations per all of the discussion above.

I will set up the repository this morning, doubt I will get much farther than that today.

OK, perfect. I will go for making typocomath somewhat less extensive, driving toward the polynomial solver to replicate the benchmark, and try to include a matrix-lite class called Vector which can be either homogeneous or inhomogeneous. (The plan there is to make it a template `Vector<T>` but _allow_ `T` to be `any` to cover the inhomogeneous case; in pocomath, substitution of `any` for the type parameter was specifically disallowed. I will try for it still to be disallowed by default, since I still don't think `Complex<Any>` makes sense, but with an escape mechanism to specifically permit it for a specific template like Vector.) I won't try to grab any of the implementation of Matrix, but for the rest I will try to stick to the mathjs implementations except if as you say there seems to be something really worthwhile that the architecture allows. Finally, I will try as a first pass to organize the implementations of the operations per all of the discussion above. I will set up the repository this morning, doubt I will get much farther than that today.

josdejong commented 2022-12-02 14:08:51 +00:00

Copy Link

👍

The plan there is to make it a template Vector<T> but allow T to be any to cover the inhomogeneous case

👌 that sounds perfect and future proof

👍 > The plan there is to make it a template `Vector<T>` but _allow_ `T` to be `any` to cover the inhomogeneous case 👌 that sounds perfect and future proof

glen referenced this issue from glen/typocomath 2022-12-02 16:06:24 +00:00

How to specify dependencies non-redundantly? #1

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

benchmark

polynomialRoot

return_types

template_types

template_operations

Labels

Clear labels   

Design

Notes, issues, or decisions related to the system design.

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

example

An item to be implemented by way of demoing a capability

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

wontfix

This won't be fixed

No Label

Design

bug

duplicate

enhancement

example

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: glen/pocomath#55

No description provided.