Trajectory for Typocomath #10

New Issue

glen · 2023-08-21T21:02:05Z

glen commented

2023-08-21 21:02:05 +00:00

As we discus the way forward, some questions that should be considered (not at all exhaustive)

Are we going for full TypeScript typing throughout?

I think we reached the conclusion that we can try for a full TypeScript codebase, but that possibly in the innards of Dispatcher we might use very "loose" types and cast back to the specific type in the results of calls.

How do we get at the "typed-functions"?

Was the design that after you constructed a Dispatcher from a bunch of "installSpecifications" (i.e., the implementations collected from all the modules), it would then have a method like dispatcher.functions() that returned a plain object all of whose properties were the appropriately assembled, properly-typed functions, e.g.

let dispatcher = new Dispatcher(allSpecifications);
let math = dispatcher.functions();
console.log('Product of 5 and 4 is`, math.multiply(5,4))

Perhaps most importantly, what should the TypeScript type of square be?

At the moment, square has just one implementation, which is generic say with type parameter T, and depends on multiply: {a:T, b:T} => T. Do we want the square function generated by Dispatcher to also be generic (say with type parameter T) and then take a single parameter of type T and return a result of type T? That will lead to a runtime type error if you execute square('foo') as there is no multiply: (a:string, b:string) => string implementation. So that seems "un-TypeScript-y". Or do we want the code to figure out all of the legal instantiations of the type T where there is a valid multiply, and type square to only take those types? That seems more true to TypeScript but also more work.

As we discus the way forward, some questions that should be considered (not at all exhaustive) 1) Are we going for full TypeScript typing throughout? I think we reached the conclusion that we can try for a full TypeScript codebase, but that possibly in the innards of Dispatcher we might use very "loose" types and cast back to the specific type in the results of calls. 2) How do we get at the "typed-functions"? Was the design that after you constructed a Dispatcher from a bunch of "installSpecifications" (i.e., the implementations collected from all the modules), it would then have a method like `dispatcher.functions()` that returned a plain object all of whose properties were the appropriately assembled, properly-typed functions, e.g. ``` let dispatcher = new Dispatcher(allSpecifications); let math = dispatcher.functions(); console.log('Product of 5 and 4 is`, math.multiply(5,4)) ``` 3) Perhaps most importantly, what should the TypeScript type of `square` be? At the moment, square has just one implementation, which is generic say with type parameter T, and depends on `multiply: {a:T, b:T} => T`. Do we want the `square` function generated by Dispatcher to also be generic (say with type parameter T) and then take a single parameter of type T and return a result of type T? That will lead to a runtime type error if you execute `square('foo')` as there is no `multiply: (a:string, b:string) => string` implementation. So that seems "un-TypeScript-y". Or do we want the code to figure out all of the legal instantiations of the type T where there is a valid multiply, and type square to only take those types? That seems more true to TypeScript but also more work.

glen commented

2023-08-22 05:15:23 +00:00

Thinking about it a bit more, here is another question:

What should the relationship between mathjs/typocomath types and TypeScript types be and how should they be specified?

As of this writing, each "math type" corresponds to a specific TypeScript type. To use it in the math system, you need to

(A) add to the AssociatedTypes interface an object that maps the math system name of the type to the TypeScript type and to various related types (its one, zero, nan, and real types), AND

(B) export a symbol that must be called

`${name}_type`

(where name is the math system name of the type) whose value specifies a predicate that checks whether an unknown entity is that type (typically a TypeScript type guard the underlying TypeScript type, but this is not enforced, I don't think), and any automatic conversions to that math type, and optionally the math system name of a type that the dispatcher should be sure this type is tested for before (to allow more specific types to be handled differently than more general ones), and optionally for generic types a method for inferring the math system type name of the parameter type from an entity that is generically of this type.

Is this correspondence 1-1 enough between math system types and TypeScript types to avoid confusions?

It seems decent on this score to me, except the two different aspects of specifying the type (exporting stuff in the AssociatedTypes interface and exporting an identifier with a conventional name) seems a bit clunky to me. Especially, having different formats/semantics for exported identifiers (either implementations or type specifications) depending on whether they end in _type seems pretty hacky and certainly complicates the code in the Dispatcher constructor. I understand where this "dual registration" of types comes from: the TypeScript-type-system information associated with the type goes in the AssociatedTypes interface, and the operational stuff (functions and strings that are needed) goes in the Foo_type export.

Is there some way to clean this up and/or make the association between TypeScript types and math system types crisper?

For example, it occurs to me that the before property, if it ends up being used, is a string literal, so it could be specified in AssociatedTypes. So what remains is the operations that test membership in the type, convert to the type, and infer the name of the parameter type for generics. It occurs to me that the parameter type inference could be folded into type testing; rather than have it return boolean, it could return the fully qualified type name or the falsy '' if the entity is not of the given type. And so it seems to me that instead of exporting this special structure with the name Foo_type to "register" type Foo, we could instead insist there was an exported operation of type (a: unknown) => string named isFoo and if any automatic conversions to type Foo are desired there should be an exported operation named autoFoo all of whose implementations are unary; any parameter type accepted by any implementation would be automatically converted to Foo. We might also arrange that all implementations for autoFoo are automatically added to an operation called Foo, along with any implementations exported explicitly for Foo -- those are the non-automatic conversions/constructions that can be invoked by calling math.Foo(x), or e.g. math.Complex(re, im).

So going down this path, a math system type would be precisely any TypeScript type Foo for which AssociatedTypes resolved to the proper type information, and for which an isFoo operation is exported. Note for this to work, the system would need to know (at both compile time and runtime, I think) the "name" of the math system type. For consistency's sake, it seems as though it had better be the string produced by the runtime type reflection system (whatever we are using) for the underlying TypeScript type. And I think there there need to be TypeScript type level operations that will go from type to the string literal type of the name and vice versa, as well as a runtime functions that will go from an entity to its type name. Those things are missing in the current state of typocomath -- there is a placeholder for the runtime function, typeOf, but it is not implemented.

Lest you think these musings are all academic, I found issue #11 by trying to figure out how we might streamline the typocomath "type system" and bring it closer to TypeScript's built-in type system (not that we wouldn't have eventually found it anyway.)

Returning to this point, it's not quite clear to me how we can (A) check/enforce that the keys in the AssociatedTypes interface match the ones returned by the runtime type reflection system and (B) compute that name from a type at both compile time and runtime. Those things need figuring out in any case...

Thinking about it a bit more, here is another question: 4. What should the relationship between mathjs/typocomath types and TypeScript types be and how should they be specified? As of this writing, each "math type" corresponds to a specific TypeScript type. To use it in the math system, you need to (A) add to the AssociatedTypes interface an object that maps the math system name of the type to the TypeScript type and to various related types (its one, zero, nan, and real types), AND (B) export a symbol that must be called ``` `${name}_type` ``` (where `name` is the math system name of the type) whose value specifies a predicate that checks whether an unknown entity is that type (typically a TypeScript type guard the underlying TypeScript type, but this is not enforced, I don't think), and any automatic conversions to that math type, and optionally the math system name of a type that the dispatcher should be sure this type is tested for before (to allow more specific types to be handled differently than more general ones), and optionally for generic types a method for inferring the math system type name of the parameter type from an entity that is generically of this type. Is this correspondence 1-1 enough between math system types and TypeScript types to avoid confusions? It seems decent on this score to me, except the two different aspects of specifying the type (exporting stuff in the AssociatedTypes interface _and_ exporting an identifier with a conventional name) seems a bit clunky to me. Especially, having different formats/semantics for exported identifiers (either implementations or type specifications) depending on whether they end in `_type` seems pretty hacky and certainly complicates the code in the Dispatcher constructor. I understand where this "dual registration" of types comes from: the TypeScript-type-system information associated with the type goes in the AssociatedTypes interface, and the operational stuff (functions and strings that are needed) goes in the `Foo_type` export. Is there some way to clean this up and/or make the association between TypeScript types and math system types crisper? For example, it occurs to me that the `before` property, if it ends up being used, is a string literal, so it could be specified in AssociatedTypes. So what remains is the operations that test membership in the type, convert to the type, and infer the name of the parameter type for generics. It occurs to me that the parameter type inference could be folded into type testing; rather than have it return boolean, it could return the fully qualified type name or the falsy `''` if the entity is not of the given type. And so it seems to me that instead of exporting this special structure with the name Foo_type to "register" type Foo, we could instead insist there was an exported operation of type `(a: unknown) => string` named `isFoo` and if any automatic conversions to type Foo are desired there should be an exported operation named `autoFoo` all of whose implementations are unary; any parameter type accepted by any implementation would be automatically converted to Foo. We might also arrange that all implementations for autoFoo are automatically added to an operation called Foo, along with any implementations exported explicitly for Foo -- those are the non-automatic conversions/constructions that can be invoked by calling `math.Foo(x)`, or e.g. `math.Complex(re, im)`. So going down this path, a math system type would be precisely any TypeScript type Foo for which AssociatedTypes<Foo> resolved to the proper type information, and for which an `isFoo` operation is exported. Note for this to work, the system would need to know (at both compile time and runtime, I think) the "name" of the math system type. For consistency's sake, it seems as though it had better be the string produced by the runtime type reflection system (whatever we are using) for the underlying TypeScript type. And I think there there need to be TypeScript type level operations that will go from type to the string literal type of the name and vice versa, as well as a runtime functions that will go from an entity to its type name. Those things are missing in the current state of typocomath -- there is a placeholder for the runtime function, `typeOf`, but it is not implemented. Lest you think these musings are all academic, I found issue #11 by trying to figure out how we might streamline the typocomath "type system" and bring it closer to TypeScript's built-in type system (not that we wouldn't have eventually found it anyway.) Returning to this point, it's not quite clear to me how we can (A) check/enforce that the keys in the AssociatedTypes interface match the ones returned by the runtime type reflection system and (B) compute that name from a type at both compile time and runtime. Those things need figuring out in any case...

glen commented

2023-08-22 06:18:55 +00:00

This last question inspires
5) (How) can each type specify its own "read syntax" for the parser?

If part of the point of the TypeScript rewrite is to allow the easy addition of types, then ideally as much information about the type is decoupled into the module for that type as possible. The representation for displaying a type is easy to specify, for example, it could just be an implementation for the operation string taking one parameter of that type. (There will likely be some further elaboration of this, such as different formats for displaying numbers, etc.) But what about in the formula parser for specifying an item of the type? Right now in mathjs, the parser has deep in its guts things like parseDoubleQuoteString and parseObject and parseNumber and parseMatrix. So these are built-in types whose literal syntax are separately built in to the parser, and if one wanted to add a type like Point with notation <2.5, -1.3> (presuming that doesn't conflict with existing expressions, of which I am not sure) I think you're pretty much out of luck without also modifying the internal parser code. Conversely, if you have not bundled matrices into your bundle, then ideally the matrix notation should not be recognized. (Frankly I am not sure what happens right now in mathjs as it stands, I am not really 100% clear how you make a bundle with everything except matrices. If you do, I guess it will parse a matrix expression into a parse tree that includes ArrayNodes, and the error will come when you try to compile that parse tree -- it won't know how to compile an ArrayNode. But to me it would seem to make more sense in a bundle without Matrix or Array for math.parse([1, 2, 3]) to throw a syntax error. )

Ideally there would be a way for each type to have a separate recognizer/reader, and once the parser got down to the case of an individual literal entity, it could call all the recognizers in turn, presumably in the same order that it checks an entity for what type it is. If the parser is going to more or less remain the same as it is now, I think this relates to the cascade of parse methods that commences with parseCustomNodes. Inside there, the parsing is pretty much really just a list of cases (implemented by calls that test a condition and if the condition holds, they return a parse, otherwise they just call the next possibility in the chain). So that whole cascade could be replaced by a loop over different recognizers, checking if each one in turn matches the current state. And those recognizers could be delegated to the various types, for example via operations named like recognizeFoo. For this perhaps the ParseState would have to be bundled up as a type and allowed as a parameter type of operations. And I guess many types will have to define their own associated node types. It seems possible this mechanism could replace the CustomNodes facility altogether -- if you want to parse some special thing, make a type for it and implement the recognizer.

This last question inspires 5) (How) can each type specify its own "read syntax" for the parser? If part of the point of the TypeScript rewrite is to allow the easy addition of types, then ideally as much information about the type is decoupled into the module for that type as possible. The representation for displaying a type is easy to specify, for example, it could just be an implementation for the operation `string` taking one parameter of that type. (There will likely be some further elaboration of this, such as different formats for displaying numbers, etc.) But what about in the formula parser for specifying an item of the type? Right now in mathjs, the parser has deep in its guts things like parseDoubleQuoteString and parseObject and parseNumber and parseMatrix. So these are built-in types whose literal syntax are separately built in to the parser, and if one wanted to add a type like Point with notation `<2.5, -1.3>` (presuming that doesn't conflict with existing expressions, of which I am not sure) I think you're pretty much out of luck without also modifying the internal parser code. Conversely, if you have not bundled matrices into your bundle, then ideally the matrix notation should not be recognized. (Frankly I am not sure what happens right now in mathjs as it stands, I am not really 100% clear how you make a bundle with everything except matrices. If you do, I guess it will parse a matrix expression into a parse tree that includes ArrayNodes, and the error will come when you try to compile that parse tree -- it won't know how to compile an ArrayNode. But to me it would seem to make more sense in a bundle without Matrix or Array for `math.parse([1, 2, 3])` to throw a syntax error. ) Ideally there would be a way for each type to have a separate recognizer/reader, and once the parser got down to the case of an individual literal entity, it could call all the recognizers in turn, presumably in the same order that it checks an entity for what type it is. If the parser is going to more or less remain the same as it is now, I think this relates to the cascade of parse methods that commences with parseCustomNodes. Inside there, the parsing is pretty much really just a list of cases (implemented by calls that test a condition and if the condition holds, they return a parse, otherwise they just call the next possibility in the chain). So that whole cascade could be replaced by a loop over different recognizers, checking if each one in turn matches the current state. And those recognizers could be delegated to the various types, for example via operations named like `recognizeFoo`. For this perhaps the ParseState would have to be bundled up as a type and allowed as a parameter type of operations. And I guess many types will have to define their own associated node types. It seems possible this mechanism could replace the CustomNodes facility altogether -- if you want to parse some special thing, make a type for it and implement the recognizer.

glen commented

2023-08-22 17:43:28 +00:00

Report on discussion between Glen and Jos on Aug 22. The numbering is on the questions as above in this issue.

We confirmed that the current goal is to have strict TypeScript typing in the implementations, loose typing in the innards, and the ability to generated detailed .d.ts files in a build step that restores strict typing for a user of the resulting math bundle.
The proposed scheme for generating the operations from the dispatcher seemed fine.
We agreed that in a module that creates a Dispatcher and generates its functions object like const math = myDispatcher.functions() on the fly, the type that the TypeScript compiler computes at compile time for math.square may well be something loose like <T>(a: T) => T that will accept any type, but that the dispatcher will have another method myDispatcher.declarations() that produces the text of a .d.ts file such that if this math object is exported and then used in association with the generated .d.ts file, not the one that TypeScript wrote out itself, then math.square will correctly have the type (a: number) => number | (a: Complex<number>) => Complex<number> | ... (etc. for all of the types T that have a multiply(a:T, b: T) operation
We agreed for now that math system types will be just TypeScript types. So we won't at the moment try to support NumInt or the like that TypeScript doesn't really support. (Maybe with a sufficiently powerful runtime typer they could be added in the future). We agreed that such types must be explicitly "registered" for use with the math system, and that should be done by exporting a symbol with the necessary information, of which the key item is the type guard. The "registration symbol" should be recognized not by a syntactic condition on its name (currently having suffix _type) but instead by a semantic condition on its value.

Although we didn't discuss it, that "registration symbol" can't be the name of the type: typically (like for Complex), the module defining the type will have to export a definition for the type itself. You can't export a type and an object with the same name in a module. So it seems like we should just leave the registration symbol and its value as they are, nothing wrong with e.g. Complex_type, just switch to recognizing it on a semantic rather than syntactic condition so there is nothing "magic" about names ending in _type and no real requirement that such names be used, just a convention.

Specifying "read syntax"?

We did not discuss this one, seems like an issue that will be left until we are implementing a parser in this framework.

New questions we discussed:

Will we allow inhomogeneous collections or just homogeneous ones, like Matrix<number>?

Three possibilities arose here:

(A) have both homogeneous and inhomogeneous ones, and possibly a class that could do either depending on whether it happens to be constructed with all elements of matching type;

(B) Have Matrix be only homogeneous and Array be inhomogeneous;

(C) write only homogeneous code, and then support inhomogeneous ones by e.g. Matrix<any> or Matrix<unknown> (not sure which would be more appropriate), with the proviso that we beef up the implementation selector so that when you ask for the implementation handling a parameter of type any or unknown, you get the full top-level runtime-dispatching function that handles all registered types.

I think the consensus was to head toward (C) for now and see how it works out. This begged question:

Will any or unknown, whichever we decide is more appropriate, be a first-class type in the system, so that it is legal to provide implementations that operate on arguments of that type? Or will they be sort of second-tier types, in that they can only be used when deduced, e.g. if you somehow manage to make a Complex with a number real part and a bigint imaginary part, it becomes a Complex<unknown> and addition has to do full dispatch, etc.

This was unclear, we decided to punt for now. In practice, I will proceed without implementing anything explicit for unknown or any and see how things fall out.

Since this has gotten long, I will put a summary of planned actions in a next comment.

Report on discussion between Glen and Jos on Aug 22. The numbering is on the questions as above in this issue. 1) We confirmed that the current goal is to have strict TypeScript typing in the implementations, loose typing in the innards, and the ability to generated detailed `.d.ts` files in a build step that restores strict typing for a user of the resulting math bundle. 2) The proposed scheme for generating the operations from the dispatcher seemed fine. 3) We agreed that in a module that creates a Dispatcher and generates its functions object like `const math = myDispatcher.functions()` on the fly, the type that the TypeScript compiler computes at compile time for `math.square` may well be something loose like `<T>(a: T) => T` that will accept any type, but that the dispatcher will have another method `myDispatcher.declarations()` that produces the text of a `.d.ts` file such that if this math object is exported and then used in association with the _generated_ `.d.ts` file, not the one that TypeScript wrote out itself, then `math.square` will correctly have the type `(a: number) => number | (a: Complex<number>) => Complex<number> | ...` (etc. for all of the types T that have a `multiply(a:T, b: T)` operation 4) We agreed for now that math system types will be just TypeScript types. So we won't at the moment try to support NumInt or the like that TypeScript doesn't really support. (Maybe with a sufficiently powerful runtime typer they could be added in the future). We agreed that such types must be explicitly "registered" for use with the math system, and that should be done by exporting a symbol with the necessary information, of which the key item is the type guard. The "registration symbol" should be recognized not by a syntactic condition on its name (currently having suffix `_type`) but instead by a semantic condition on its value. Although we didn't discuss it, that "registration symbol" can't be the name of the type: typically (like for Complex), the module defining the type will have to export a definition for the type itself. You can't export a type and an object with the same name in a module. So it seems like we should just leave the registration symbol and its value as they are, nothing wrong with e.g. `Complex_type`, just switch to recognizing it on a semantic rather than syntactic condition so there is nothing "magic" about names ending in `_type` and no real requirement that such names be used, just a convention. 5) Specifying "read syntax"? We did not discuss this one, seems like an issue that will be left until we are implementing a parser in this framework. New questions we discussed: 6) Will we allow inhomogeneous collections or just homogeneous ones, like `Matrix<number>`? Three possibilities arose here: (A) have both homogeneous and inhomogeneous ones, and possibly a class that could do either depending on whether it happens to be constructed with all elements of matching type; (B) Have Matrix be only homogeneous and Array be inhomogeneous; (C) write only homogeneous code, and then support inhomogeneous ones by e.g. `Matrix<any>` or `Matrix<unknown>` (not sure which would be more appropriate), with the proviso that we beef up the implementation selector so that when you ask for the implementation handling a parameter of type `any` or `unknown`, you get the full top-level runtime-dispatching function that handles all registered types. I think the consensus was to head toward (C) for now and see how it works out. This begged question: 7) Will `any` or `unknown`, whichever we decide is more appropriate, be a first-class type in the system, so that it is legal to provide implementations that operate on arguments of that type? Or will they be sort of second-tier types, in that they can only be used when deduced, e.g. if you somehow manage to make a Complex with a number real part and a bigint imaginary part, it becomes a `Complex<unknown>` and addition has to do full dispatch, etc. This was unclear, we decided to punt for now. In practice, I will proceed without implementing anything explicit for `unknown` or `any` and see how things fall out. Since this has gotten long, I will put a summary of planned actions in a next comment.

glen commented

2023-08-22 17:56:36 +00:00

Planned actions based on the discussion:

Jos:

Will perform experiments with (a) template literal type description allowing inferred strong typing of implementations with a runtime structure describing the type available -- Glen tried but lost type information in the deduction, eliminating the major point of type safety in implementations.
and with (b) writing our own TypeScript plugin
and with (c) getting ts-macros to run. Note we need to watch for new releases of ts-macros because they will have fixes we need (type deduction not depending on argument order, for example) and features we may want (such as exporting within a macro)
Then Jos will decide which runtime typing method we will go with for the full prototype, (a), (b), or (c)

Glen:

Will fix issues filed that don't relate to runtime typing in main
When those are done, if Jos is not yet done with the runtime typing selection, Glen will work on top of the ts-macros branch to begin to develop the "full prototype". This work can be migrated to a different runtime typer if (a) or (b) is selected.

What does the "full prototype" mean? We agreed that the target is a typocomath with all of the features of the pocomath proof-of-concept working except non-TypeScript types like NumInt plus the ability to write out .d.ts files. So the next step after outstanding issues are resolved is to stop faking the installations of implementations and actually start generating some instantiated operations.

Planned actions based on the discussion: Jos: - Will perform experiments with (a) template literal type description allowing inferred strong typing of implementations with a runtime structure describing the type available -- Glen tried but lost type information in the deduction, eliminating the major point of type safety in implementations. - and with (b) writing our own TypeScript plugin - and with (c) getting ts-macros to run. Note we need to watch for new releases of ts-macros because they will have fixes we need (type deduction not depending on argument order, for example) and features we may want (such as exporting within a macro) - Then Jos will decide which runtime typing method we will go with for the full prototype, (a), (b), or (c) Glen: - Will fix issues filed that don't relate to runtime typing in main - When those are done, if Jos is not yet done with the runtime typing selection, Glen will work on top of the ts-macros branch to begin to develop the "full prototype". This work can be migrated to a different runtime typer if (a) or (b) is selected. What does the "full prototype" mean? We agreed that the target is a typocomath with all of the features of the pocomath proof-of-concept working _except_ non-TypeScript types like NumInt _plus_ the ability to write out .d.ts files. So the next step after outstanding issues are resolved is to stop faking the installations of implementations and actually start generating some instantiated operations.

glen referenced this issue

2023-08-22 17:58:03 +00:00

Discriminate symbols registering types semantically, not syntactically #12

glen referenced this issue

2023-08-30 14:54:21 +00:00

Inferring TypeScript types #5

josdejong referenced this issue

2023-08-30 14:57:10 +00:00

Inferring TypeScript types #5

josdejong commented

2023-09-01 12:51:08 +00:00

Thanks, very clear summary!

(4) I think we can define something like the following. The dispatcher must be able to recognize whether it is importing a function or a type definition:

export interface MathjsTypeDefinition {
  name: string
  test: ...
  from: ...
}
export function isMathjsTypeDefinition (value) {...}

export type Complex<T> = { re: T; im: T; }
export function isComplex(value) {...}

export interface ComplexTypeDefinition extends MathjsTypeDefinition {
  name: 'Complex',
  test: isComplex,
  from: ...
}

(5) The parser is currently hardcoded. We can refactor it to make it more flexible and allow inserting new parts in the parser to handle new notations and types. I think that this topic can be handled separately from this core refactor of mathjs. I think it will be good to keep the data types unaware of the existence of an expression parser in order not to overload them. But we could indeed think through a "plugin" solution for the expression parser. That way we could create a module DataTypeX that comes with (a) the data type (b) a parser plugin and (c) a set of basic functions like add, subtract, etc.

(6) agree, let's optimize for homogenious matrices like Matrix<number> (that is the common case and optimizing can improve the performance a lot) and allow Matrix<unknown> or Matrix<T> too, more as a second class citizen.

Thanks, very clear summary! (4) I think we can define something like the following. The dispatcher must be able to recognize whether it is importing a function or a type definition: ```ts export interface MathjsTypeDefinition { name: string test: ... from: ... } export function isMathjsTypeDefinition (value) {...} export type Complex<T> = { re: T; im: T; } export function isComplex(value) {...} export interface ComplexTypeDefinition extends MathjsTypeDefinition { name: 'Complex', test: isComplex, from: ... } ``` (5) The parser is currently hardcoded. We can refactor it to make it more flexible and allow inserting new parts in the parser to handle new notations and types. I think that this topic can be handled separately from this core refactor of mathjs. I think it will be good to keep the data types unaware of the existence of an expression parser in order not to overload them. But we could indeed think through a "plugin" solution for the expression parser. That way we could create a module DataTypeX that comes with (a) the data type (b) a parser plugin and (c) a set of basic functions like add, subtract, etc. (6) agree, let's optimize for homogenious matrices like `Matrix<number>` (that is the common case and optimizing can improve the performance a lot) and allow `Matrix<unknown>` or `Matrix<T>` too, more as a second class citizen.

josdejong commented

2023-09-01 12:58:08 +00:00

I'll be working on the defined experiments (template literals, typescript plugin, ts-macros) this afternoon and next week Thursday, and hope that we can make a decision end of next week.

josdejong commented

2023-09-01 16:48:17 +00:00

I just implemented my first TypeScript plugin, and it replaces all occurrences of the string '__infer__' with the actual types of a function 😎

It's in the experiment/typescript_plugin branch, and I explained the details in the readme:

https://code.studioinfinity.org/glen/typocomath/src/branch/experiment/typescript_plugin

If you strip all the comments and console.logs from the plugin, the plugin contains just a few lines of actual code.

I just implemented my first TypeScript plugin, and it replaces all occurrences of the string `'__infer__'` with the actual types of a function 😎 It's in the `experiment/typescript_plugin` branch, and I explained the details in the readme: https://code.studioinfinity.org/glen/typocomath/src/branch/experiment/typescript_plugin If you strip all the comments and console.logs from the plugin, the plugin contains just a few lines of actual code.

glen commented

2023-09-01 22:14:50 +00:00

Well, this is an impressive start! Congrats! If you really want to go down this path, rather than just work with an externally-maintained package like ts-macros, then I think the next steps/questions are:

(A) See what happens when the types are specified with the generics we want to use (like Dependencies<'multiply', T>) rather than explicitly written out. Can you resolve those generics to the same explicit form? I did get that working with ts-macros with some additional generic trickery...

(B) Why is the name 'square' being reiterated from the identifier square in the experiment/arithmeticInfer.ts? Is that needed for this experiment to work? There is no such duplication in src/generic/arithmetic.ts.

(C) Devise some automated way to get these type annotations attached to (all? or if not, which?) function entities so that square can simply be exported without needing to write typed('__infer__', [actual definition]) for every implementation of every operation in the whole codebase...

Seems like if we can get those three things ironed out, we will have a winner :)

Well, this is an impressive start! Congrats! If you really want to go down this path, rather than just work with an externally-maintained package like ts-macros, then I think the next steps/questions are: (A) See what happens when the types are specified with the generics we want to use (like `Dependencies<'multiply', T>`) rather than explicitly written out. Can you resolve those generics to the same explicit form? I did get that working with ts-macros with some additional generic trickery... (B) Why is the name 'square' being reiterated from the identifier `square` in the experiment/arithmeticInfer.ts? Is that needed for this experiment to work? There is no such duplication in src/generic/arithmetic.ts. (C) Devise some automated way to get these type annotations attached to (all? or if not, which?) function entities so that `square` can simply be exported without needing to write `typed('__infer__', [actual definition])` for every implementation of every operation in the whole codebase... Seems like if we can get those three things ironed out, we will have a winner :)

glen commented

2023-09-02 02:28:04 +00:00

Oh, realized there was another message here I hadn't responded to.

(4) The dispatcher must be able to recognize whether it is importing a function or a type definition:

This is already done in main.

(5) The parser is currently hardcoded. We can refactor it to make it more flexible and allow inserting new parts in the parser to handle new notations and types. I think that this topic can be handled separately from this core refactor of mathjs. I think it will be good to keep the data types unaware of the existence of an expression parser in order not to overload them. But we could indeed think through a "plugin" solution for the expression parser. That way we could create a module DataTypeX that comes with (a) the data type (b) a parser plugin and (c) a set of basic functions like add, subtract, etc.

I agree that the actual classes Complex<T> etc. should not have parsing code attached to them. But I think we also agree that parsing specific to a datatype has to be located in the directory or module for that datatype -- it's the only way I can think of to fully decouple types from each other. We don't want the parser saying "Hey there should be a matrix here" in a bundle that has no matrices, for example.

(6) agree, let's optimize for homogenious matrices like Matrix<number> (that is the common case and optimizing can improve the performance a lot) and allow Matrix<unknown> or Matrix<T> too, more as a second class citizen.

Oh, realized there was another message here I hadn't responded to. > (4) The dispatcher must be able to recognize whether it is importing a function or a type definition: This is already done in main. > (5) The parser is currently hardcoded. We can refactor it to make it more flexible and allow inserting new parts in the parser to handle new notations and types. I think that this topic can be handled separately from this core refactor of mathjs. I think it will be good to keep the data types unaware of the existence of an expression parser in order not to overload them. But we could indeed think through a "plugin" solution for the expression parser. That way we could create a module DataTypeX that comes with (a) the data type (b) a parser plugin and (c) a set of basic functions like add, subtract, etc. I agree that the actual classes `Complex<T>` etc. should not have parsing code attached to them. But I think we also agree that parsing specific to a datatype has to be located in the directory or module for that datatype -- it's the only way I can think of to fully decouple types from each other. We don't want the parser saying "Hey there should be a matrix here" in a bundle that has no matrices, for example. > (6) agree, let's optimize for homogenious matrices like `Matrix<number>` (that is the common case and optimizing can improve the performance a lot) and allow `Matrix<unknown>` or `Matrix<T>` too, more as a second class citizen.

josdejong commented

2023-09-06 10:30:20 +00:00

Yes, it looks like a simple home-made TS plugin may just do the job 😅

(A) I'll give that a try tomorrow, good to verify that.
(B) There is no need for that, I was just trying out to create something typed-function-like. I'll remove it.
(C) That would be awesome indeed to automatically infer the functions that need it. But how can the plugin know what functions in the code must be inferred? We need some way to recognize that at compile time (both when using ts-macros or our own plugin).

I'll do more playing around with it tomorrow.

(4) 👍
(5) agree, it makes most sense to put parser extensions for a data type in the same "module" as that datatype

Yes, it looks like a simple home-made TS plugin may just do the job 😅 (A) I'll give that a try tomorrow, good to verify that. (B) There is no need for that, I was just trying out to create something typed-function-like. I'll remove it. (C) That would be awesome indeed to automatically infer the functions that need it. But how can the plugin know what functions in the code must be inferred? We need _some_ way to recognize that at compile time (both when using `ts-macros` or our own plugin). I'll do more playing around with it tomorrow. (4) 👍 (5) agree, it makes most sense to put parser extensions for a data type in the same "module" as that datatype

glen commented

2023-09-07 15:16:01 +00:00

(C) That would be awesome indeed to automatically infer the functions that need it. But how can the plugin know what functions in the code must be inferred? We need some way to recognize that at compile time (both when using ts-macros or our own plugin).

I mean, I was thinking of some indicator, like a special function call that is a no-op but is a signal to the plugin when it is present, that types should be inferred and attached as a property to every exported function in that source file. Something along those lines. It's OK if we annotate some functions we don't end up needing the types of. So just some brief "flag" to turn the type generation on, to keep all of the rest of the syntax ordinary and concise.

> (C) That would be awesome indeed to automatically infer the functions that need it. But how can the plugin know what functions in the code must be inferred? We need _some_ way to recognize that at compile time (both when using `ts-macros` or our own plugin). I mean, I was thinking of some indicator, like a special function call that is a no-op but is a signal to the plugin when it is present, that types should be inferred and attached as a property to every exported function in that source file. Something along those lines. It's OK if we annotate some functions we don't end up needing the types of. So just some brief "flag" to turn the type generation on, to keep all of the rest of the syntax ordinary and concise.

josdejong commented

2023-09-08 14:29:19 +00:00

yes indeed. I'm changing it to be something like the following, matching $reflect (or whatever name we want):

export const square = $reflect(<T>(dep: Dependencies<'multiply', T>): (a: T) => T =>
  z => dep.multiply(z, z)
)

I'm still figuring out how to get Dependencies<'multiply', T> resolved, that is not magically working but I expect that the type checker has a method for that.

yes indeed. I'm changing it to be something like the following, matching `$reflect` (or whatever name we want): ```ts export const square = $reflect(<T>(dep: Dependencies<'multiply', T>): (a: T) => T => z => dep.multiply(z, z) ) ``` I'm still figuring out how to get `Dependencies<'multiply', T>` resolved, that is not magically working but I expect that the type checker has a method for that.

glen commented

2023-09-08 15:30:28 +00:00

yes indeed. I'm changing it to be something like the following, matching $reflect (or whatever name we want):

Hmm, I was thinking more like a compiler directive:

$reflectExports(); // now every function exported from this file will be tagged with its type

export function square<T>(dep: Dependencies<'multiply', T>) {
   return (z: T) => dep.multiply(z,z)
}

Seems like that will lead to much less useless repetition of $reflect(...)

Note also that in this example I have used a function rather than arrow notation here. It seems to me that should be allowed; at the moment typocomath is written only in arrow notation, but looking at it now it doesn't seem to me like that's a requirement, unless I am not remembering something. Also it seems as though sometimes we can rely on TypeScript's type inference (although I haven't tested this, so I am not sure!!); note I elided the explicit annotation Signature<'square', T> and explicitly annotated z as type T to give inference something to work on. That elision could possibly also work in arrow notation:

export const square = <T>(dep: Dependencies<'multiply', T>) => (z: T) => dep.multiply(z,z)

> yes indeed. I'm changing it to be something like the following, matching `$reflect` (or whatever name we want): Hmm, I was thinking more like a compiler directive: ``` $reflectExports(); // now every function exported from this file will be tagged with its type export function square<T>(dep: Dependencies<'multiply', T>) { return (z: T) => dep.multiply(z,z) } ``` Seems like that will lead to much less useless repetition of `$reflect(...)` Note also that in this example I have used a function rather than arrow notation here. It seems to me that should be allowed; at the moment typocomath is written only in arrow notation, but looking at it now it doesn't seem to me like that's a requirement, unless I am not remembering something. Also it seems as though sometimes we can rely on TypeScript's type inference (although I haven't tested this, so I am not sure!!); note I elided the explicit annotation `Signature<'square', T>` and explicitly annotated `z` as type `T` to give inference something to work on. That elision could possibly also work in arrow notation: ``` export const square = <T>(dep: Dependencies<'multiply', T>) => (z: T) => dep.multiply(z,z) ```

josdejong commented

2023-09-11 16:22:56 +00:00

I've done some more experimenting with a plugin of our own. I didn't manage to get the resolved types out of it. I know it is possible (using functionsl getTypeAtLocation, getResolvedSignature, and typeToString), but don't want to spend too much time on that now and rather implement the prototype using ts-macros.

I still want to give the template literal type a little try but I don't have much expectations there.

I managed to get the ts-macros branch running on my computer, but for some reason it only works with ttypescript and not with typescript. I've pushed the changes needed to get it running on my Windows machine into the ts-macros-jos branch for now.

I've done some more experimenting with a plugin of our own. I didn't manage to get the resolved types out of it. I know it is possible (using functionsl `getTypeAtLocation`, `getResolvedSignature`, and `typeToString`), but don't want to spend too much time on that now and rather implement the prototype using `ts-macros`. I still want to give the template literal type a little try but I don't have much expectations there. I managed to get the `ts-macros` branch running on my computer, but for some reason it only works with `ttypescript` and not with `typescript`. I've pushed the changes needed to get it running on my Windows machine into the `ts-macros-jos` branch for now.

glen commented

2023-09-11 16:27:27 +00:00

I managed to get the ts-macros branch running on my computer, but for some reason it only works with ttypescript and not with typescript. I've pushed the changes needed to get it running on my Windows machine into the ts-macros-jos branch for now.

Huh, even though you did the

after pnpm install, you must execute npx ts-patch install to activate the ts-macros compiler plugin.

from the README in that branch? Weird, I wish I could help debug that, but I am not sure how.

Just let me know when you've settled on a scheme to go forward with and we can merge it into main so development can go forward. Thanks for all the efforts!

> I managed to get the `ts-macros` branch running on my computer, but for some reason it only works with `ttypescript` and not with `typescript`. I've pushed the changes needed to get it running on my Windows machine into the `ts-macros-jos` branch for now. > Huh, even though you did the > after pnpm install, you must execute `npx ts-patch install` to activate the ts-macros compiler plugin. from the README in that branch? Weird, I wish I could help debug that, but I am not sure how. Just let me know when you've settled on a scheme to go forward with and we can merge it into main so development can go forward. Thanks for all the efforts!

josdejong commented

2023-09-11 16:40:49 +00:00

Huh, even though you did the

after pnpm install, you must execute npx ts-patch install to activate the ts-macros compiler plugin.

O wow, that's just it... I feel a bit stupid, ha ha.

Ok if I update the cp in the pnpm scripts to something that works on both windows and linux?

> Huh, even though you did the > > > after pnpm install, you must execute `npx ts-patch install` to activate the ts-macros compiler plugin. > O wow, that's just it... I feel a bit stupid, ha ha. Ok if I update the `cp` in the pnpm scripts to something that works on both windows and linux?

glen commented

2023-09-11 16:48:00 +00:00

Ok if I update the cp in the pnpm scripts to something that works on both windows and linux?

Of course, but you shouldn't need to. See https://pnpm.io/cli/run#shell-emulator

I thought I had checked in a .npmrc file that did this, but apparently not -- my apologies. Anyhow, give it a whirl and if it works, one or the other of us can supply the appropriate .npmrc. Thanks.

> Ok if I update the `cp` in the pnpm scripts to something that works on both windows and linux? Of course, but you shouldn't need to. See https://pnpm.io/cli/run#shell-emulator I thought I had checked in a `.npmrc` file that did this, but apparently not -- my apologies. Anyhow, give it a whirl and if it works, one or the other of us can supply the appropriate .npmrc. Thanks.

josdejong commented

2023-09-15 07:36:37 +00:00

As discussed here, I've done some more experimenting last weeks:

(A) I've experimented more with template literal types, see #7 (comment). I got a step further, but still a long way to go. This approach is really inflexible (if we get it to work at all) and I think that inflexibility would hurt us. So, let us not pursue that path further for now.
(B) Build our own TypeScript compiler. I'm quite sure we can make it work, but don't want to spend more time on it now: we can use ts-macros which will do about the same.
(C) Got ts-macros running on my machine. This approach is quite flexible and straightforward, it feels like a solid approach.

So, for the TypeScript based POC for mathjs, I propose that we use ts-macros and get going now 😄. The goal is to get a TS based POC that is functionally the same as the JS based pocomath. Once we have that, we should have a clear view on whether TS actually helps us or hurts us, and then decide whether we go for TS or JS for the source code of mathjsNext. What do you think Glen?

As [discussed here](https://code.studioinfinity.org/glen/typocomath/issues/10#issuecomment-1030), I've done some more experimenting last weeks: - (A) I've experimented more with template literal types, see https://code.studioinfinity.org/glen/typocomath/issues/7#issuecomment-1242. I got a step further, but still a long way to go. This approach is really inflexible (if we get it to work at all) and I think that inflexibility would hurt us. So, let us not pursue that path further for now. - (B) Build our own TypeScript compiler. I'm quite sure we can make it work, but don't want to spend more time on it now: we can use `ts-macros` which will do about the same. - (C) Got `ts-macros` running on my machine. This approach is quite flexible and straightforward, it feels like a solid approach. So, for the TypeScript based POC for mathjs, I propose that we use `ts-macros` and get going now 😄. The goal is to get a TS based POC that is functionally the same as the JS based [`pocomath`](https://code.studioinfinity.org/glen/pocomath/). Once we have that, we should have a clear view on whether TS actually helps us or hurts us, and then decide whether we go for TS or JS for the source code of mathjsNext. What do you think Glen?

glen commented

2023-09-15 14:19:43 +00:00

Totally fine with that and will start working that way. But a couple of caveats:

doing the prototype this way likely locks us in to either (B) or (C) for the "real thing" if we decide to go the typescript way. Are you OK with that?
if we go the typescript way I would be leaving any possible implementation of (B) to you, I don't have any interest/energy for diving into the compiler internals. Are you OK with that?

And then I have a pragmatic question: when ts-macros is working OK, is it OK to merge it into main here, or do you prefer to just stay on a ts-macros branch indefinitely in case we decide to revist A/B/C later?

Totally fine with that and will start working that way. But a couple of caveats: - doing the prototype this way likely locks us in to either (B) or (C) for the "real thing" if we decide to go the typescript way. Are you OK with that? - if we go the typescript way I would be leaving any possible implementation of (B) to you, I don't have any interest/energy for diving into the compiler internals. Are you OK with that? And then I have a pragmatic question: when ts-macros is working OK, is it OK to merge it into main here, or do you prefer to just stay on a ts-macros branch indefinitely in case we decide to revist A/B/C later?

josdejong commented

2023-09-15 15:18:05 +00:00

Well, we have to decide, right? The ts-macros approach all in all looks most promising to me for a TS approach for mathjs. Still, the goal of the POC is to validate whether it works out nicely. I hope so, but if not, we will need to rethink it again.

For now let's not look further into our own typescript plugin (B), but I'm totally fine with looking further into that if we see value in it. A refactor from (C) to (B) will be quite straightforward since they are very similar in approach, so I don't see a big risk there.

Let's start implementing the POC in the main branch of typocomath and get it on par with the functionality in pocomath? I would like to keep all branches with all the experiments there though and not throw them away, OK?

Well, we have to decide, right? The `ts-macros` approach all in all looks most promising to me for a TS approach for mathjs. Still, the goal of the POC is to _validate_ whether it works out nicely. I hope so, but if not, we will need to rethink it again. For now let's not look further into our own typescript plugin (B), but I'm totally fine with looking further into that if we see value in it. A refactor from (C) to (B) will be quite straightforward since they are very similar in approach, so I don't see a big risk there. Let's start implementing the POC in the `main` branch of `typocomath` and get it on par with the functionality in `pocomath`? I would like to keep all branches with all the experiments there though and not throw them away, OK?

glen commented

2023-09-15 15:51:44 +00:00

Sounds like a plan. Next step: I will update the ts-macros release in the branch and narrow it down to one particular macro that we can use, and make sure it seems to work in a couple of places. When that's done I will pause for you to review that the syntax is looking OK from your point of view. When we are in agreement I will merge ts-macros into main so we can proceed with getting the POC up to the same level of actual operation as pocomath. Throughout, I will keep the experimental branches as they are.

Thanks for the discussion! I hope to get to the point where I said "I will pause" by Monday, but y'know, nothing is certain...

Sounds like a plan. Next step: I will update the ts-macros release in the branch and narrow it down to one particular macro that we can use, and make sure it seems to work in a couple of places. When that's done I will pause for you to review that the syntax is looking OK from your point of view. When we are in agreement I will merge ts-macros into main so we can proceed with getting the POC up to the same level of actual operation as pocomath. Throughout, I will keep the experimental branches as they are. Thanks for the discussion! I hope to get to the point where I said "I will pause" by Monday, but y'know, nothing is certain...

josdejong commented

2023-09-15 16:19:12 +00:00

💪

glen referenced this issue

2023-09-21 20:22:30 +00:00

Inferring TypeScript types #5

josdejong referenced this issue

2023-09-22 07:45:34 +00:00

Inferring TypeScript types #5

josdejong commented

2023-09-22 07:47:32 +00:00

Continuation from #5 (comment)...

Great! That's a nice chunk of work. If you're comfortable just moving ahead with ts-macros for type reflection and putting the other two alternatives aside for the time being, then that's cool with me.

Yes indeed, I'm quite happy with that! As for the alternatives, (like discussed in #10 (comment)): I have little hope for string literal types to work out in the end, and a custom typescript plugin will work more or less the same as ts-macros, so no fundamental difference in how the POC will work in the end. So yeah I would like to focus on ts-macros and see where we end up.

Don't understand why this wasn't working -- I thought one of the points of pnpm was it provided a cross-platform scripting environment -- but this is fine too, the point is just to get things working.

I could very well do something wrong myself, but right now I just want the setup to work one way or another and focus on the actual POC, we can figure it out later.

Yes good for testing but should we wait until there's a properly-published npm package before merging into main?

That is fine with me, ts-macros@2.4.2 should be published within a few days.

Nice! Output looks promising. Needs some minor touch-up like clearly indicating whether the operation is generic and if so what the template parameter is (I think we are limiting to one parameter for now). Or maybe we are just assuming the parameter is always T?

I think for the POC it is enough to be a minimalistic here and require it to be always T. There is no technical hurdle in interpreting a differently named generic I think.

Continuation from https://code.studioinfinity.org/glen/typocomath/issues/5#issuecomment-1295... > Great! That's a nice chunk of work. If you're comfortable just moving ahead with ts-macros for type reflection and putting the other two alternatives aside for the time being, then that's cool with me. Yes indeed, I'm quite happy with that! As for the alternatives, (like discussed in https://code.studioinfinity.org/glen/typocomath/issues/10#issuecomment-1250): I have little hope for string literal types to work out in the end, and a custom typescript plugin will work more or less the same as `ts-macros`, so no fundamental difference in how the POC will work in the end. So yeah I would like to focus on `ts-macros` and see where we end up. > Don't understand why this wasn't working -- I thought one of the points of pnpm was it provided a cross-platform scripting environment -- but this is fine too, the point is just to get things working. I could very well do something wrong myself, but right now I just want the setup to work one way or another and focus on the actual POC, we can figure it out later. > Yes good for testing but should we wait until there's a properly-published npm package before merging into main? That is fine with me, `ts-macros@2.4.2` should be published within a few days. > Nice! Output looks promising. Needs some minor touch-up like clearly indicating whether the operation is generic and if so what the template parameter is (I think we are limiting to one parameter for now). Or maybe we are just assuming the parameter is always T? I think for the POC it is enough to be a minimalistic here and require it to be always T. There is no technical hurdle in interpreting a differently named generic I think.

josdejong commented

2023-09-22 08:06:43 +00:00

I think by far the biggest piece is writing the guts of the dispatcher.

We will indeed have to rewrite the dispatcher in a solution that has the typed-function functionality integrated/merged. I'm not sure though if we need to do that as part of this TypeScript POC. Do you think we can postpone this step and first get a TS version working of pocomath? If we are not happy with how the TS POC works out in the end, we may consider going just JS after all. I would like to spend as little effort as possible to validate the TS approach, so we can make this decision as early as possible.

With the TypeScript POC I want to validate:

That a TS approach (with ts-macros) is really feasible: make it work
See how the developer experience will be, and whether the TS integration works smooth or not when:
- I as a mathjs developer want to implement a new function in mathjs
- I as a mathjs consumer want to use mathjs as-is
- I as a mathjs consumer want to extend the library with a new data type or function

For example the developer experience is not yet ideal in typocomath right now: when developing, the files with $implement functions do not actually export anything since that is handled by the macro at compile time, so it looks like nothing is exported and my IDE gives errors there:

This is something that I would like to think though and solve ASAP, this is a bad TS experience. It may be not a problem when the only one consuming these files is internally in the dispatcher, I'm not entirely sure. But it probably helps already if the files export a minimal typescript definition like sqrt: Function rather than nothing. I expect that we come across more of these kinds of issues when working out the TS POC.

> I think by far the biggest piece is writing the guts of the dispatcher. We will indeed have to rewrite the dispatcher in a solution that has the typed-function functionality integrated/merged. I'm not sure though if we need to do that as part of this TypeScript POC. Do you think we can postpone this step and first get a TS version working of `pocomath`? If we are not happy with how the TS POC works out in the end, we may consider going just JS after all. I would like to spend as little effort as possible to validate the TS approach, so we can make this decision as early as possible. With the TypeScript POC I want to validate: 1. That a TS approach (with `ts-macros`) is really feasible: make it work 2. See how the developer experience will be, and whether the TS integration works smooth or not when: - I as a mathjs developer want to implement a new function in mathjs - I as a mathjs consumer want to use mathjs as-is - I as a mathjs consumer want to extend the library with a new data type or function For example the developer experience is not yet ideal in `typocomath` right now: when developing, the files with `$implement` functions do not actually export anything since that is handled by the macro at compile time, so it looks like nothing is exported and my IDE gives errors there: ![afbeelding](/attachments/66fb7571-fb2e-4106-bcf2-c557950dc7e9) This is something that I would like to think though and solve ASAP, this is a bad TS experience. It may be not a problem when the only one consuming these files is internally in the dispatcher, I'm not entirely sure. But it probably helps already if the files export a minimal typescript definition like `sqrt: Function` rather than nothing. I expect that we come across more of these kinds of issues when working out the TS POC.

afbeelding.png

30 KiB

glen commented

2023-09-22 15:35:29 +00:00

Nice! Output looks promising. Needs some minor touch-up like clearly indicating whether the operation is generic and if so what the template parameter is (I think we are limiting to one parameter for now). Or maybe we are just assuming the parameter is always T?

I think for the POC it is enough to be a minimalistic here and require it to be always T. There is no technical hurdle in interpreting a differently named generic I think.

OK, that's fine. In that case, I would just add a flag to the parsed type as to whether it was generic or specific. I can do that when I next work on it.

> > Nice! Output looks promising. Needs some minor touch-up like clearly indicating whether the operation is generic and if so what the template parameter is (I think we are limiting to one parameter for now). Or maybe we are just assuming the parameter is always T? > I think for the POC it is enough to be a minimalistic here and require it to be always T. There is no technical hurdle in interpreting a differently named generic I think. OK, that's fine. In that case, I would just add a flag to the parsed type as to whether it was generic or specific. I can do that when I next work on it.

👍 1

glen commented

2023-09-22 16:16:57 +00:00

We will indeed have to rewrite the dispatcher in a solution that has the typed-function functionality integrated/merged. I'm not sure though if we need to do that as part of this TypeScript POC. Do you think we can postpone this step and first get a TS version working of pocomath?

I am confused: pocomath only works because of a core which consists of actual current typed-function with a very complicated/crufty layer (it was just a POC) on top of it that adds more features like generics, early resolution of dependencies, and subtypes. We want typocomath at least to support TypeScript generics and early resolution, so that needs implementing, and I am not sure there will be much value in trying to use existing typed-function for the little that's left that it has to offer... In any case, there's a fair bit of machinery in there that has to work to get a "TS version working of pocomath" and although I think we can try to skip the subtypes stuff at least for now, which will reduce complexity noticeably, I personally am not seeing any shortcut to getting something running that avoids writing a new Dispatcher...

For example the developer experience is not yet ideal in typocomath right now: when developing, the files with $implement functions do not actually export anything since that is handled by the macro at compile time, so it looks like nothing is exported and my IDE gives errors there:

Oh that's weird, but it is also to some extent my fault. Since I just write code in an editor and compile frequently rather than look at IDE annotations (probably not the most efficient, I know, but I am a creature of long habit), I always just want to make the most compact notation. So that's what I did with $implement. I just assumed IDEs used tsc under the hood and the patched tsc would give the correct language services. But I guess I was overoptimistic.

I can change the macro so that we write either

export const square = $reflect!(<T>(dep: Dependencies<'multiply',T>) => (z: T) => dep.multiply(z,z))

or

export function square<T>(dep: Dependencies<'multiply', T>) {
   return (z:T) => dep.multiply(z,z)
}
$reflect!('square');

(Note in the second format, the definition of square could be an arrow function, too, I didn't mean to imply it would have to be a function.)
Which one of these should I try? My guess is the IDE will be happiest with the second, but it does require us to reiterate the implementation name...)

Let me know and I will make that my next priority. I can do it in ts-macros-issues even before the next release of ts-macros drops.

> We will indeed have to rewrite the dispatcher in a solution that has the typed-function functionality integrated/merged. I'm not sure though if we need to do that as part of this TypeScript POC. Do you think we can postpone this step and first get a TS version working of pocomath? I am confused: pocomath only works because of a core which consists of actual current typed-function with a very complicated/crufty layer (it was just a POC) on top of it that adds more features like generics, early resolution of dependencies, and subtypes. We want typocomath at least to support TypeScript generics and early resolution, so that needs implementing, and I am not sure there will be much value in trying to use existing typed-function for the little that's left that it has to offer... In any case, there's a fair bit of machinery in there that has to work to get a "TS version working of pocomath" and although I _think_ we can try to skip the subtypes stuff at least for now, which will reduce complexity noticeably, I personally am not seeing any shortcut to getting something running that avoids writing a new Dispatcher... >For example the developer experience is not yet ideal in typocomath right now: when developing, the files with $implement functions do not actually export anything since that is handled by the macro at compile time, so it looks like nothing is exported and my IDE gives errors there: Oh that's weird, but it is also to some extent my fault. Since I just write code in an editor and compile frequently rather than look at IDE annotations (probably not the most efficient, I know, but I am a creature of long habit), I always just want to make the most compact notation. So that's what I did with $implement. I just assumed IDEs used tsc under the hood and the patched tsc would give the correct language services. But I guess I was overoptimistic. I can change the macro so that we write either ``` export const square = $reflect!(<T>(dep: Dependencies<'multiply',T>) => (z: T) => dep.multiply(z,z)) ``` or ``` export function square<T>(dep: Dependencies<'multiply', T>) { return (z:T) => dep.multiply(z,z) } $reflect!('square'); ``` (Note in the second format, the definition of square could be an arrow function, too, I didn't mean to imply it would have to be a `function`.) Which one of these should I try? My guess is the IDE will be happiest with the second, but it does require us to reiterate the implementation name...) Let me know and I will make that my next priority. I can do it in ts-macros-issues even before the next release of ts-macros drops.

josdejong commented

2023-09-22 17:56:00 +00:00

O sorry, I was under the assumption that we can leave the core of the pocomath Dispatcher more or less as-is for the typocomath POC. You've written the core, so you know best and if you say it will be necessary to rewrite it for the typocomath POC I believe you :). I just would like to take as much shortcuts in the POC phase as reasonably possible (like skipping subtypes for now if possible).

I think export const square = $reflect!(...) can work if we can define a type like $reflect<T>(factory: T) : T. I have to say that I do like your second format a lot too! It completely separates the regular TS code from the "macro magic", and that will probably work best with IDE's and all kinds of tools. It would be even nicer if it can be defined on top, I'm now having a annotation/decorator style in mind that would look very natural to many programmers (not sure if we can do that with ts-macros, just a thought right now):

@reflect('square')
export function square<T>(dep: Dependencies<'multiply', T>) {
   return (z:T) => dep.multiply(z,z)
}

O sorry, I was under the assumption that we can leave the core of the `pocomath` Dispatcher more or less as-is for the `typocomath` POC. You've written the core, so you know best and if you say it will be necessary to rewrite it for the `typocomath` POC I believe you :). I just would like to take as much shortcuts in the POC phase as reasonably possible (like skipping subtypes for now if possible). I think `export const square = $reflect!(...)` can work if we can define a type like `$reflect<T>(factory: T) : T`. I have to say that I do like your second format a lot too! It completely separates the regular TS code from the "macro magic", and that will probably work best with IDE's and all kinds of tools. It would be even nicer if it can be defined on top, I'm now having a annotation/decorator style in mind that would look very natural to many programmers (not sure if we can do that with `ts-macros`, just a thought right now): ```ts @reflect('square') export function square<T>(dep: Dependencies<'multiply', T>) { return (z:T) => dep.multiply(z,z) } ```

glen commented

2023-09-22 18:22:09 +00:00

Yes, we can skip subtypes for now as far as I know. Hopefully they will not really come up since we agreed to stick with types that can correspond 1-1 to TypeScript types, which means we will be dropping "numint" from typocomath as compared to pocomath, and that's the main part where subtypes were used. (This is sort of ironic in that TypeScript of course has full support for subtypes, but I don't see that we will be using them in the POC. They could always be added later if we get everything else working).

On the format for reflect: JavaScript, and hence TypeScript and ts-macros does not allow decorators on ordinary functions. But since function and variable declarations are hoisted anyway, I think it should work to write

$reflect!('square')
export const square = <T>(dep: Dependencies<'multiply', T>)  =>
   (z:T) => dep.multiply(z,z)

so I will try for that format. Should have at least that bit done by Monday.

Yes, we can skip subtypes for now as far as I know. Hopefully they will not really come up since we agreed to stick with types that can correspond 1-1 to TypeScript types, which means we will be dropping "numint" from typocomath as compared to pocomath, and that's the main part where subtypes were used. (This is sort of ironic in that TypeScript of course has full support for subtypes, but I don't see that we will be using them in the POC. They could always be added later if we get everything else working). On the format for `reflect`: JavaScript, and hence TypeScript and ts-macros does not allow decorators on ordinary functions. But since function and variable declarations are hoisted anyway, I think it should work to write ``` $reflect!('square') export const square = <T>(dep: Dependencies<'multiply', T>) => (z:T) => dep.multiply(z,z) ``` so I will try for that format. Should have at least that bit done by Monday.

👍 1

josdejong commented

2023-09-29 13:42:49 +00:00

I did two small steps:

upgrade to ts-macros@2.5.0 instead of using the git source. Ok if I merge ts-macros-issues into main and then delete ts-macros-issues and ts-macros?
I refactored the macro from:
```
$implement!('sqrt', (dep) => {...})
```
into:
```
export const sqrt = $reflect!('sqrt', (dep) => {...})
```
And let $reflect() return Impl & { reflectedType: string} so TS is happy everywhere in my IDE too, solving this issue.

I didn't manage to put it as a separate decorator like syntax like you proposed, that would be neat though. Do you think that is possible? I saw the docs on Decorator Macros of ts-macros, but TS decorators are only supported with classes and methods, not with functions or constants.

I did two small steps: 1. upgrade to `ts-macros@2.5.0` instead of using the git source. Ok if I merge `ts-macros-issues` into `main` and then delete `ts-macros-issues` and `ts-macros`? 2. I refactored the macro from: ```ts $implement!('sqrt', (dep) => {...}) ``` into: ```ts export const sqrt = $reflect!('sqrt', (dep) => {...}) ``` And let `$reflect()` return `Impl & { reflectedType: string}` so TS is happy everywhere in my IDE too, solving [this issue](https://code.studioinfinity.org/glen/typocomath/issues/10#issuecomment-1302). I didn't manage to put it as a separate decorator like syntax like you proposed, that would be neat though. Do you think that is possible? I saw the docs on [Decorator Macros](https://github.com/GoogleFeud/ts-macros/wiki/Decorator-Macros) of ts-macros, but TS decorators are only supported with classes and methods, not with functions or constants.

glen commented

2023-09-30 23:57:00 +00:00

Ok if I merge ts-macros-issues into main and then delete ts-macros-issues and ts-macros?

Sure, that's fine. Or I can once (2) below is settled.

(syntax of the macro)
I pushed another commit to ts-macros-issues in which I provided another alternative syntax:

export const sqrt = (dep) => {
   // body goes here
}
$reflecType(sqrt)

That way the code part is absolutely clean, there's just this annotation after that we want to reflect its type. Unfortunately I too could not get the "reflect directive" to come before the function -- if you just move this definition above, it violates the "can't use before declaration" rule. Or, if you make the exported function a "var" instead of a "const", the generation works fine, but then the property 'reflectedType" doesn't show up in the exported type, i.e. I get a

src/index.ts:30:66 - error TS2339: Property 'reflectedType' does not exist on type ...

error. But since that is just a type error, we could patch that up with a bit of TypeScript machination. So if you really prefer, we could switch to

$reflecType(sqrt)
export var sqrt = (dep) => {
   // body goes here
}

I also tried to make a macro that would reflect a whole bunch of types at once, but couldn't get it to work, see https://github.com/GoogleFeud/ts-macros/issues/78

Feel free to pick any of these notations that work and make it so, or just let me know.

1) > Ok if I merge ts-macros-issues into main and then delete ts-macros-issues and ts-macros? Sure, that's fine. Or I can once (2) below is settled. 2) (syntax of the macro) I pushed another commit to ts-macros-issues in which I provided another alternative syntax: ``` export const sqrt = (dep) => { // body goes here } $reflecType(sqrt) ``` That way the code part is absolutely clean, there's just this annotation after that we want to reflect its type. Unfortunately I too could not get the "reflect directive" to come before the function -- if you just move this definition above, it violates the "can't use before declaration" rule. Or, if you make the exported function a "var" instead of a "const", the generation works fine, but then the property 'reflectedType" doesn't show up in the exported type, i.e. I get a ``` src/index.ts:30:66 - error TS2339: Property 'reflectedType' does not exist on type ... ``` error. But since that is just a type error, we could patch that up with a bit of TypeScript machination. So if you really prefer, we could switch to ``` $reflecType(sqrt) export var sqrt = (dep) => { // body goes here } ``` I also tried to make a macro that would reflect a whole bunch of types at once, but couldn't get it to work, see https://github.com/GoogleFeud/ts-macros/issues/78 Feel free to pick any of these notations that work and make it so, or just let me know.

glen commented

2023-10-01 16:42:35 +00:00

GoogleFeud has already responded to the issue I created and referenced in the previous post. Apparently it wasn't implemented, but will be in the next version. So if we wait for that, we should be able to do

export const add = ... // exactly the code we would use without reflection
export const subtract = ... // etc.
export const multiply = ...
export const divide = ...
$reflecTypes!([add, subtract, multiply, divide])

I think personally I like that the best as being the most compact, but as I said I am also fine with the notation you implemented

export const sqrt = $reflect!(...)

or the one I just implemented

export const conservativeSqrt = ...
$reflect!(conservativeSqrt)

or if you like the directive before, with a little type annotation tweak in Dispatcher we could do:

$reflect!(square)
export var square = ...

[Note var rather than const to allow mention before declaration.]

As I said in the last post, just let me know which one you decide on and I or you can make it so (and merge into main, as far as I am concerned).

Thanks!

GoogleFeud has already responded to the issue I created and referenced in the previous post. Apparently it wasn't implemented, but will be in the next version. So if we wait for that, we should be able to do ``` export const add = ... // exactly the code we would use without reflection export const subtract = ... // etc. export const multiply = ... export const divide = ... $reflecTypes!([add, subtract, multiply, divide]) ``` I think personally I like that the best as being the most compact, but as I said I am also fine with the notation you implemented ``` export const sqrt = $reflect!(...) ``` or the one I just implemented ``` export const conservativeSqrt = ... $reflect!(conservativeSqrt) ``` or if you like the directive before, with a little type annotation tweak in Dispatcher we could do: ``` $reflect!(square) export var square = ... ``` [Note `var` rather than `const` to allow mention before declaration.] As I said in the last post, just let me know which one you decide on and I or you can make it so (and merge into main, as far as I am concerned). Thanks!

josdejong commented

2023-10-04 13:31:28 +00:00

I really like to separate the $reflect from the "clean" TS function. I also like your idea of invoking $reflect with multiple functions in one go, that will be very handy and compact in files with a set of one-line functions! Great to hear that that will be supported by ts-macros.

I have a slight preference for defining the $reflect macro above the function rather than below, but not at the cost of going to use var or introducing other complexities. I think for the POC it is fine to define it below the function. If we are not happy with how that works out, we can dive deeper (I expect we can solve that if we implement our own TypeScript plugin, but I don't want to go there now).

There is one small issue with your latest version of $reflect!: TypeScript is not aware (in the IDE) that the function is extended with a property .reflectedType. Can we somehow extend the types of the reflected functions with that (like in my previous version of $reflect it would extend the function definition with & { reflectedType: string }). It is not a deal-breaker to me though: this property will only be used internally by our Dispatcher, and is normally not needed for end users.

I really like to separate the `$reflect` from the "clean" TS function. I also like your idea of invoking `$reflect` with multiple functions in one go, that will be very handy and compact in files with a set of one-line functions! Great to hear that that will be supported by ts-macros. I have a slight preference for defining the `$reflect` macro above the function rather than below, but not at the cost of going to use `var` or introducing other complexities. I think for the POC it is fine to define it below the function. If we are not happy with how that works out, we can dive deeper (I expect we can solve that if we implement our own TypeScript plugin, but I don't want to go there now). There is one small issue with your latest version of `$reflect!`: TypeScript is not aware (in the IDE) that the function is extended with a property `.reflectedType`. Can we somehow extend the types of the reflected functions with that (like in my previous version of $reflect it would extend the function definition with ` & { reflectedType: string }`). It is not a deal-breaker to me though: this property will only be used internally by our Dispatcher, and is normally not needed for end users.

glen commented

2023-10-04 14:40:24 +00:00

Right, I wasn't sure whether .reflectedType being or not being on the symbol for the IDE was actually a benefit or a detriment...

Anyhow, it's only the IDE that gets confused; tsc knows the property is there and can access it in dispatcher with no difficulty.

I am completely unfamiliar with how an IDE derives type info, but I sort of doubt it can be convinced to change the type of an identifier by calling a function on it, since that's usually impossible.

So given that, do you want to
(A) go back to export square = $reflect!('square', DEF HERE)
(B) move ahead with the reflect-the-types-of-multiple-identifiers-at-the-end-of-the-file function?

Thanks for letting me know.

Right, I wasn't sure whether .reflectedType being or not being on the symbol for the IDE was actually a benefit or a detriment... Anyhow, it's only the IDE that gets confused; tsc knows the property is there and can access it in dispatcher with no difficulty. I am completely unfamiliar with how an IDE derives type info, but I sort of doubt it can be convinced to change the type of an identifier by calling a function on it, since that's usually impossible. So given that, do you want to (A) go back to `export square = $reflect!('square', DEF HERE)` (B) move ahead with the reflect-the-types-of-multiple-identifiers-at-the-end-of-the-file function? Thanks for letting me know.

josdejong commented

2023-10-04 14:54:13 +00:00

I say let's continue with your latest approach:

export const sqrt = ...
$reflect!(sqrt)

I'll see if I can add this & { reflectedType: string } type information somehow, but it is a nice-to-have and no big problem.

I say let's continue with your latest approach: ```ts export const sqrt = ... $reflect!(sqrt) ``` I'll see if I can add this ` & { reflectedType: string }` type information somehow, but it is a nice-to-have and no big problem.

glen commented

2023-10-04 15:30:42 +00:00

OK, I'll merge that into main as soon as I get through the eigenvector stuff.

👍 1

glen commented

2023-10-10 04:40:59 +00:00

OK, I updated to the latest version of ts-macros and got a several-reflections-at-once macro $reflecTypes working (don't worry, if you don't like this slightly silly name, happy to pick something else). That's pushed to the ts-macros-issues branch.

However, I have not yet pushed this to main, because there turns out to be yet one more hitch. The $reflectType(s) macros are just adding lines of code like
foo.reflectedType = "(deps: {add: (a: number, b: number) => number}) => (a: number) => number"

which is fine and gives foo the type of the function that was originally assigned to it together with a string reflectedType property, if foo was not explicitly typed at declaration.

However, almost all of the implementation exports currently have explicit types, e.g.

export const add: Signature<'add', number> = (a, b) => a + b;

in order to add explicit typechecking that the given implementation conforms to the declared signature for an 'add' function on numbers, as opposed to just the presumption that the given formula has the correct type.
In the presence of this explicit typing, the generated code causes an error TS2339: Property 'reflectedType' does not exist on type '(a: number, b: number) => number'. So to get pnpm go to work in this branch, I had to comment out the : Signature<'add', number> part of the above line.

I see three possible ways around this irritation:

(1) Go ahead and remove all explicit typings of implementations. The resulting code export const add = (a: number, b: number) => a+b is still pretty readable. We lose TypeScript doing signature consistency checks, i.e., we could then define an 'add' implementation of three booleans producing a string, say, which would not conform to any declared signature of 'add', but since the full type is reflected, we could put in code in the Dispatcher that would recover such checking, although it would then take place later in the build-execute cycle. Early inconsistency detection is a good thing...

(2) Modify the generic type Signature<operation, paramType> (and possibly some of its friends) to allow an optional reflectedType property, or

(3) Switch back to a macro syntax like

export const add = $reflect!<Signature<'add', number>>((a, b) => a + b)

and (try to) put the "right" type declaration/checking code into the macro expansion.

But there may be other possibilities besides these three that I am not thinking of. Of these three, since I like the compactness of the "reflect several identifiers at once at the end of the file", I guess I lean toward (2).

So before we move on, do you want to make a choice of (1), (2), (3), or some better other alternative I didn't come up with? Thanks!

OK, I updated to the latest version of ts-macros and got a several-reflections-at-once macro `$reflecTypes` working (don't worry, if you don't like this slightly silly name, happy to pick something else). That's pushed to the ts-macros-issues branch. However, I have not yet pushed this to main, because there turns out to be yet one more hitch. The $reflectType(s) macros are just adding lines of code like `foo.reflectedType = "(deps: {add: (a: number, b: number) => number}) => (a: number) => number"` which is fine and gives foo the type of the function that was originally assigned to it together with a string reflectedType property, **if foo was not explicitly typed at declaration**. However, almost all of the implementation exports currently have explicit types, e.g. `export const add: Signature<'add', number> = (a, b) => a + b;` in order to add explicit typechecking that the given implementation conforms to the declared signature for an 'add' function on numbers, as opposed to just the presumption that the given formula has the correct type. In the presence of this explicit typing, the generated code causes an `error TS2339: Property 'reflectedType' does not exist on type '(a: number, b: number) => number'.` So to get `pnpm go` to work in this branch, I had to comment out the `: Signature<'add', number>` part of the above line. I see three possible ways around this irritation: (1) Go ahead and remove all explicit typings of implementations. The resulting code `export const add = (a: number, b: number) => a+b` is still pretty readable. We lose TypeScript doing signature consistency checks, i.e., we could then define an 'add' implementation of three booleans producing a string, say, which would not conform to any declared signature of 'add', but since the full type is reflected, we could put in code in the Dispatcher that would recover such checking, although it would then take place later in the build-execute cycle. Early inconsistency detection is a good thing... (2) Modify the generic type `Signature<operation, paramType>` (and possibly some of its friends) to allow an optional reflectedType property, or (3) Switch back to a macro syntax like `export const add = $reflect!<Signature<'add', number>>((a, b) => a + b)` and (try to) put the "right" type declaration/checking code into the macro expansion. But there may be other possibilities besides these three that I am not thinking of. Of these three, since I like the compactness of the "reflect several identifiers at once at the end of the file", I guess I lean toward (2). So before we move on, do you want to make a choice of (1), (2), (3), or some better other alternative I didn't come up with? Thanks!

josdejong commented

2023-10-11 08:38:31 +00:00

😂 $reflecTypes. I expect it will lead to confusion though, it looks like a typo, so we have to change it to $reflectTypes in the end I think. But totally fine with me to keep it in for now to have some fun 😉. Thinking aloud: would it be possible to use the same $reflect for both a single signature and multiple signatures? I'm not sure if we should do that, having explicit functions is often better than overloading a single function with a lot of functionality.

I like option 2 the most too. The .reflectedType property is mostly for internal use, I think it's a pragmatic solution to solve it like this, and better than the alternatives I think.

😂 `$reflecTypes`. I expect it will lead to confusion though, it looks like a typo, so we have to change it to `$reflectTypes` in the end I think. But totally fine with me to keep it in for now to have some fun 😉. Thinking aloud: would it be possible to use the same `$reflect` for both a single signature and multiple signatures? I'm not sure if we should do that, having explicit functions is often better than overloading a single function with a lot of functionality. I like option 2 the most too. The `.reflectedType` property is mostly for internal use, I think it's a pragmatic solution to solve it like this, and better than the alternatives I think.

glen commented

2023-10-13 21:24:49 +00:00

OK, as soon as I get a chance, hopefully this weekend, I will modify Signature and possibly others to allow the reflectedType property -- is that what we really want to use in practice, or do we want to try to use something like a Symbol for the property name that will be more or less guaranteed to be collision-free with other libraries?

In other news, there's just now a new entry into the runtime types for TypeScript category, see https://docs.rttist.org/. Do we want to give that a whirl before we move ahead, or just stick with ts-macros?

As far as the macro names, yes, I wasn't being too serious with $reflecTypes. For serious proposals, we could just do $reflect! or I also like $runTypes! or $RTType!. Let me know if you have a preference. I think we should just stick with the form $reflect!([foo, bar, baz]) -- if you want to do just one, you can always say $reflect!([foo]) -- an extra pair of square braces, boo hoo, it's supposed to stick out in your face that we're doing something nonstandard anyway.

OK, as soon as I get a chance, hopefully this weekend, I will modify Signature and possibly others to allow the reflectedType property -- is that what we really want to use in practice, or do we want to try to use something like a Symbol for the property name that will be more or less guaranteed to be collision-free with other libraries? In other news, there's just now a new entry into the runtime types for TypeScript category, see https://docs.rttist.org/. Do we want to give that a whirl before we move ahead, or just stick with ts-macros? As far as the macro names, yes, I wasn't being too serious with $reflecTypes. For serious proposals, we could just do `$reflect!` or I also like `$runTypes!` or `$RTType!`. Let me know if you have a preference. I think we should just stick with the form `$reflect!([foo, bar, baz])` -- if you want to do just one, you can always say `$reflect!([foo])` -- an extra pair of square braces, boo hoo, it's supposed to stick out in your face that we're doing something nonstandard anyway.

glen commented

2023-10-15 16:58:04 +00:00

Alright, I looked at docs.rttist.org, and it's just a frontend on top of hookyns/tst-reflect which we also considered and rejected in our last round of trying reflection systems. So scratch that.

So to try to wrap this up, I will add a commit to ts-macros-issues that leaves everything on the .reflectedType property, adjusts our generic signature types to include that, and has a single macro $reflect! that takes an array of items to reflect. If you'd like any different choices on any of those aspects, one of us can just change it in another commit before merging back to main. I won't merge 'til you've had a chance to look at the proposed "final" version of ts-macros-issues and decide on those two questions: (1) what property for the type data, and (2) name/call structure of the macro or macros we provide.

Alright, I looked at docs.rttist.org, and it's just a frontend on top of hookyns/tst-reflect which we also considered and rejected in our last round of trying reflection systems. So scratch that. So to try to wrap this up, I will add a commit to ts-macros-issues that leaves everything on the .reflectedType property, adjusts our generic signature types to include that, and has a single macro `$reflect!` that takes an array of items to reflect. If you'd like any different choices on any of those aspects, one of us can just change it in another commit before merging back to main. I won't merge 'til you've had a chance to look at the proposed "final" version of ts-macros-issues and decide on those two questions: (1) what property for the type data, and (2) name/call structure of the macro or macros we provide.

glen commented

2023-10-16 03:41:05 +00:00

OK, well, I have done most of what I proposed in the last comment. However, there is a hitch with option (2): when you add {reflectedType?: string} to the Signature generic type, it becomes opaque to the ts-macros $$typeToString builtin, and is reported as just Signature<'add', number>, for example. So then I thought first we would just have to look up the signatures in the Signatures interface; but I couldn't figure out how to get the full generic information out of that interface, since you have to instantiate Signatures in order to call $$typeToString on it...

Note this problem only affects implementations that have no dependencies. The ones with dependencies all work fine.

So I came up with three possible workarounds for the implementations with no dependencies. There is one working example of each in src/numbers/arithmetic.ts. My possible ideas are:

(A) Export a type RTT (for Run Time Typeable) and just write an explicit intersection with RTT for all no-dependency implementations (see 'add' in number/arithmetic.ts), rather than sticking RTT inside the generic Signature type;

(B) Write all no-dependency implementations as functions of no arguments (representing no dependencies passed in) returning the actual implementation (see 'subtract' in number/arithmetic.ts); or

(C) Make a special DSignature<> generic type (short for "Direct Signature", I'm open to another name) used only (and always) for implementations with no dependencies, and provide a matching $Dreflect!() macro for generating their type info. (see 'conj' in number/arithmetic.ts)

Of course, there may be other possibilities I haven't thought of. But we need to pick some workable plan. None of the three I came up with is as smooth as we'd like, but I don't think it matters too much, since only a very small fraction of all implementations have no dependencies. Hence I don't mind too much accepting some small extra characters or redundancy on the no-dependency implementations. And I will admit to some fatigue with this enterprise: seems like we keep hitting roadblocks. So I'd be perfectly happy if you just want to choose any one of the three options I generated, or want to come up with something else even better, so that we can get all the implementation specifications in typocomath in a working, reflectable state and try to move on to actual dispatching... I would like to see this process converge at some point. Let me know how you'd like to proceed.

(OK, I will admit that if it's all the same to you, my preferred option of the three above is (B) -- it actually makes the no-dependency implementations more parallel with the ones that have dependencies, and explicitly flags that they are dependency-free. As you may recall, my very original proposal for dependency-free implementations, like subtract on numbers, is that they should be written as

  export const subtract = () => (a: number, b: number) => a - b

i.e., give me no dependencies and I will give you back the function that subtracts two numbers. But if I recall, you didn't like the outer function with no arguments in the case of implementations with no dependencies, so maybe you won't want to choose (B). On the other hand, it looks like we are going to have to specially mark the implementations
with no dependencies in some way or other, so maybe doing it by adding that outer layer of a function of no dependency arguments isn't such a bad way. Up to you, as I said I am fine with any of A, B, C, or anything else reasonable you may come up with.)

Thanks! Sorry this is such a slog.

OK, well, I have done most of what I proposed in the last comment. However, there is a hitch with option (2): when you add `{reflectedType?: string}` to the Signature generic type, it becomes opaque to the ts-macros `$$typeToString` builtin, and is reported as just `Signature<'add', number>`, for example. So then I thought first we would just have to look up the signatures in the `Signatures` interface; but I couldn't figure out how to get the full generic information out of that interface, since you have to _instantiate_ Signatures in order to call $$typeToString on it... Note this problem only affects implementations that have no dependencies. The ones with dependencies all work fine. So I came up with three possible workarounds for the implementations with no dependencies. There is one working example of each in `src/numbers/arithmetic.ts`. My possible ideas are: (A) Export a type `RTT` (for Run Time Typeable) and just write an _explicit_ intersection with RTT for all no-dependency implementations (see 'add' in number/arithmetic.ts), rather than sticking RTT inside the generic `Signature` type; (B) Write all no-dependency implementations as functions of no arguments (representing no dependencies passed in) returning the actual implementation (see 'subtract' in number/arithmetic.ts); or (C) Make a special DSignature<> generic type (short for "Direct Signature", I'm open to another name) used only (and always) for implementations with no dependencies, and provide a matching $Dreflect!() macro for generating their type info. (see 'conj' in number/arithmetic.ts) Of course, there may be other possibilities I haven't thought of. But we need to pick some workable plan. None of the three I came up with is as smooth as we'd like, but I don't think it matters _too_ much, since only a _very_ small fraction of all implementations have no dependencies. Hence I don't mind too much accepting some small extra characters or redundancy on the no-dependency implementations. And I will admit to some fatigue with this enterprise: seems like we keep hitting roadblocks. So I'd be perfectly happy if you just want to choose any one of the three options I generated, or want to come up with something else even better, so that we can get all the implementation specifications in typocomath in a working, reflectable state and try to move on to actual dispatching... I would like to see this process converge at some point. Let me know how you'd like to proceed. (OK, I will admit that if it's all the same to you, my preferred option of the three above is (B) -- it actually makes the no-dependency implementations more parallel with the ones that have dependencies, and explicitly flags that they are dependency-free. As you may recall, my very original proposal for dependency-free implementations, like subtract on numbers, is that they should be written as ``` export const subtract = () => (a: number, b: number) => a - b ``` i.e., give me no dependencies and I will give you back the function that subtracts two numbers. But if I recall, you didn't like the outer function with no arguments in the case of implementations with no dependencies, so maybe you won't want to choose (B). On the other hand, it looks like we are going to have to specially mark the implementations with no dependencies in some way or other, so maybe doing it by adding that outer layer of a function of no dependency arguments isn't such a bad way. Up to you, as I said I am fine with any of A, B, C, or anything else reasonable you may come up with.) Thanks! Sorry this is such a slog.

josdejong commented

2023-10-16 10:35:30 +00:00

Thanks for checking out https://docs.rttist.org/. I expect that a different library can possibly only improve on syntax or compile-time performance of ts-macros. The main question we want to answer with the typocomath POC isn't about that though: it is whether we can make TS types and our own runtime type Dispatcher work together smoothly. I think ts-macros is perfectly fine for the POC since it is a working solution, and I'm happy to stick with that.

As far as the macro names: I don't have a strong opinion in that regard. I'm happy with $reflect!(foo) and $reflect!([foo, bar, baz]).

So I came up with three possible workarounds for the implementations with no dependencies.

Good point. I like your option (B) or (C) most. I think what we're (implicitly) mixing up here is whether a function is a factory function, or the actual function itself.

Your option (B) indeed looks quite straightforward, and in line with how to define functions that do have dependencies: it basically requires to write all functions as factory functions. At this stage, let's just go for option (B) and see how it works out in the end, ok? I think it is a consistent and straightforward approach, so it may work out nicely.

Just to keep in mind: there is an option (D), which is: wrap all functions themselves like we had in an earlier version: export const sqrt = $reflect!(name, (dep) => ...). But let's not go that way right now and see how option (B) works out.

I will modify Signature and possibly others to allow the reflectedType property -- is that what we really want to use in practice, or do we want to try to use something like a Symbol for the property name that will be more or less guaranteed to be collision-free with other libraries?

Good to keep that in mind indeed. I think it is just fine to attach a property reflectedType to the functions, just like we already do with attaching signatures to a typed-function. There is indeed a small collision risk. If that occurs, I think we can come up with an alternative like defining a function as an object like: {fn: Function, reflectedType: string}, which can be generated by a wrapper function like this option (D).

On a side note: in the end, I think we want to do as much work as possible during compile time, so maybe we do not want to expose .reflectedType as a string, but let it contain some read-made AST, aligned with the internal structure of the Dispatcher. This is all optimization though, and not the focus of this first POC in my opinion.

To summarize: we've been spending a lot of time coming up with the ideal syntax and tackling practical challenges to get TS types in JS at all. At this point I would love to "just" pick a reasonable syntax, and get going with the main thing: get the dispatcher working for real, and then decide whether this TS approach is the way head for mathjs or not. If we are happy, I think we can further fine-tune syntax and performance in a next stage (we want the best possible DX of course). Does that make sense?

Thanks for checking out https://docs.rttist.org/. I expect that a different library can possibly only improve on syntax or compile-time performance of `ts-macros`. The main question we want to answer with the typocomath POC isn't about that though: it is whether we can make TS types and our own runtime type Dispatcher work together smoothly. I think `ts-macros` is perfectly fine for the POC since it is a working solution, and I'm happy to stick with that. As far as the macro names: I don't have a strong opinion in that regard. I'm happy with `$reflect!(foo)` and `$reflect!([foo, bar, baz])`. > So I came up with three possible workarounds for the implementations with no dependencies. Good point. I like your option (B) or (C) most. I think what we're (implicitly) mixing up here is whether a function is a factory function, or the actual function itself. Your option (B) indeed looks quite straightforward, and in line with how to define functions that do have dependencies: it basically requires to write _all_ functions as factory functions. At this stage, let's just go for option (B) and see how it works out in the end, ok? I think it is a consistent and straightforward approach, so it may work out nicely. Just to keep in mind: there is an option (D), which is: wrap all functions themselves like we had in an earlier version: `export const sqrt = $reflect!(name, (dep) => ...)`. But let's not go that way right now and see how option (B) works out. > I will modify Signature and possibly others to allow the `reflectedType` property -- is that what we really want to use in practice, or do we want to try to use something like a Symbol for the property name that will be more or less guaranteed to be collision-free with other libraries? Good to keep that in mind indeed. I think it is just fine to attach a property `reflectedType` to the functions, just like we already do with attaching `signatures` to a `typed-function`. There is indeed a small collision risk. If that occurs, I think we can come up with an alternative like defining a function as an object like: `{fn: Function, reflectedType: string}`, which can be generated by a wrapper function like this option (D). On a side note: in the end, I think we want to do as much work as possible during compile time, so maybe we do not want to expose `.reflectedType` as a string, but let it contain some read-made AST, aligned with the internal structure of the Dispatcher. This is all optimization though, and not the focus of this first POC in my opinion. To summarize: we've been spending a lot of time coming up with the ideal syntax and tackling practical challenges to get TS types in JS at all. At this point I would love to "just" pick a reasonable syntax, and get going with the main thing: get the dispatcher working for real, and then decide whether this TS approach is the way head for mathjs or not. If we are happy, I think we can further fine-tune syntax and performance in a next stage (we want the best possible DX of course). Does that make sense?

glen commented

2023-10-17 01:24:24 +00:00

As far as the macro names: I don't have a strong opinion in that regard. I'm happy with $reflect!(foo) and $reflect!([foo, bar, baz]).

Well, at least for this iteration to keep things simple I am just going to support the array version, so that for a single one you will have to say $reflect!([foo]).

Your option (B) indeed looks quite straightforward, and in line with how to define functions that do have dependencies: it basically requires to write all functions as factory functions. At this stage, let's just go for option (B) and see how it works out in the end, ok?

Yup, I will switch everything over to that.

On a side note: in the end, I think we want to do as much work as possible during compile time, so maybe we do not want to expose .reflectedType as a string, but let it contain some read-made AST, aligned with the internal structure of the Dispatcher. This is all optimization though, and not the focus of this first POC in my opinion.

I agree; and ultimately while it would be nice to do such parsing at compile time, implementation-wise the TypeScript type language is much more cumbersome to code in, so if such pre-parsing becomes really important to performance, we might just want to use ts-macros to write out lists of signatures, and then program up an additional build step (in ordinary code) that does the parsing.

So yes, moving ahead with the current macro and option (B): make everything a factory, possibly one of no dependency arguments.

> As far as the macro names: I don't have a strong opinion in that regard. I'm happy with `$reflect!(foo)` and `$reflect!([foo, bar, baz])`. Well, at least for this iteration to keep things simple I am just going to support the array version, so that for a single one you will have to say `$reflect!([foo])`. > Your option (B) indeed looks quite straightforward, and in line with how to define functions that do have dependencies: it basically requires to write _all_ functions as factory functions. At this stage, let's just go for option (B) and see how it works out in the end, ok? Yup, I will switch everything over to that. > On a side note: in the end, I think we want to do as much work as possible during compile time, so maybe we do not want to expose `.reflectedType` as a string, but let it contain some read-made AST, aligned with the internal structure of the Dispatcher. This is all optimization though, and not the focus of this first POC in my opinion. I agree; and ultimately while it would be nice to do such parsing at compile time, implementation-wise the TypeScript type language is _much_ more cumbersome to code in, so if such pre-parsing becomes really important to performance, we might just want to use ts-macros to write out lists of signatures, and then program up an additional build step (in ordinary code) that does the parsing. So yes, moving ahead with the current macro and option (B): make everything a factory, possibly one of no dependency arguments.

👍 1

glen commented

2023-10-17 22:17:48 +00:00

OK, I finished that up and merged into main (but I did not delete branch ts-macros-issues in case we need to revisit). I enhanced the parsing so that all of the type reflections currently being generated will parse (I handled configDependency as a hack, but that may be OK since I don't think there will be more than a handful of such things).

When you get a chance, please take a look, make sure main works OK for you and everything looks reasonable for moving forward. I won't do anything else until I hear back.

After that, I think the next thing is to start implementing the Dispatcher. I think on first pass we should just make the result of constructing a Dispatcher completely closed to changes, including changing the config, since in the TypeScript world such an object can't undergo any changes that would change its interface at all, i.e. no new operations, no new types handled by existing operations, no change to return types of any operation, etc. I think ultimately we will want to allow changes to config that don't affect the signatures of any operations, and we will want to allow substituting a new implementation for one that it is completely type-compatible with, but that's all. Since those are both rarely-needed things, I think at the moment we should just make the Dispatcher completely frozen. (That will prevent any need for any code to track which operations have been invalidated by changes, which will be a big simplification in getting the Dispatcher working.) We should allow one or more Dispatchers to be among the arguments of constructing a Dispatcher, meaning to incorporate all the implementations of the existing Dispatchers into the new one, with later implementations overriding earlier ones. That will provide a way to extend/modify an existing Dispatcher (= "mathjs instance").

If you agree with that, then I think the next step is to enhance the parser to record whether the implementation was generic. Right now that info is being thrown away, and it seems better to capture it explicitly rather than just relying on whether we see the type T anywhere. I'll file an issue for that.

Looking forward to your thoughts, Glen

OK, I finished that up and merged into main (but I did not delete branch ts-macros-issues in case we need to revisit). I enhanced the parsing so that all of the type reflections currently being generated will parse (I handled configDependency as a hack, but that may be OK since I don't think there will be more than a handful of such things). When you get a chance, please take a look, make sure main works OK for you and everything looks reasonable for moving forward. I won't do anything else until I hear back. After that, I think the next thing is to start implementing the Dispatcher. I think on first pass we should just make the result of constructing a Dispatcher completely closed to changes, including changing the config, since in the TypeScript world such an object can't undergo any changes that would change its interface at all, i.e. no new operations, no new types handled by existing operations, no change to return types of any operation, etc. I think ultimately we will want to allow changes to config that don't affect the signatures of any operations, and we will want to allow substituting a new implementation for one that it is completely type-compatible with, but that's all. Since those are both rarely-needed things, I think at the moment we should just make the Dispatcher completely frozen. (That will prevent any need for any code to track which operations have been invalidated by changes, which will be a big simplification in getting the Dispatcher working.) We should allow one or more Dispatchers to be among the arguments of constructing a Dispatcher, meaning to incorporate all the implementations of the existing Dispatchers into the new one, with later implementations overriding earlier ones. That will provide a way to extend/modify an existing Dispatcher (= "mathjs instance"). If you agree with that, then I think the next step is to enhance the parser to record whether the implementation was generic. Right now that info is being thrown away, and it seems better to capture it explicitly rather than just relying on whether we see the type `T` anywhere. I'll file an issue for that. Looking forward to your thoughts, Glen

josdejong commented

2023-10-18 13:13:04 +00:00

That's good news!

I've checked out the latest version of main, and can successfully run pnpm go. There was one typo export type FunctionDef {, I've changed that to export type FunctionDef = { to make things compile. The $reflect!([...]) at the bottom looks quite neat. We'll need a clear error message when we forget to include some function there, that's definitely going to happen.

Let's take this as the base for the POC and work it out, and after that reflect on the exact syntax again.

The issue with TypeScript inside an IDE not being aware of the .reflectedType property is still there (I think we have to extend some interface with & { reflectedType: string }?):

It definitely makes to start simple with a "frozen" Dispatcher, and in the second stage make it possible to configure things. And good idea to allow creating a new dispatcher based on one or multiple existing dispatchers.

That's good news! I've checked out the latest version of `main`, and can successfully run `pnpm go`. There was one typo `export type FunctionDef {`, I've changed that to `export type FunctionDef = {` to make things compile. The `$reflect!([...])` at the bottom looks quite neat. We'll need a clear error message when we forget to include some function there, that's definitely going to happen. Let's take this as the base for the POC and work it out, and after that reflect on the exact syntax again. The issue with TypeScript inside an IDE not being aware of the `.reflectedType` property is still there (I think we have to extend some interface with ` & { reflectedType: string }`?): ![afbeelding](/attachments/9d3998c4-4d7f-4d21-880b-fb622220d8dd) It definitely makes to start simple with a "frozen" Dispatcher, and in the second stage make it possible to configure things. And good idea to allow creating a new dispatcher based on one or multiple existing dispatchers.

afbeelding.png

68 KiB

glen commented

2023-10-18 17:30:50 +00:00

I've checked out the latest version of main, and can successfully run pnpm go. There was one typo export type FunctionDef {, I've changed that to export type FunctionDef = { to make things compile.

Weird, I wonder why I didn't get any compilation error?? Thanks for fixing.

The $reflect!([...]) at the bottom looks quite neat. We'll need a clear error message when we forget to include some function there, that's definitely going to happen.

I agree, but it should also be easy to add the first time it bites one of us -- at some point we assume the reflectedType property is there, so we just check first and if it's not, issue a good message.

Let's take this as the base for the POC and work it out, and after that reflect on the exact syntax again.

Roger that.

The issue with TypeScript inside an IDE not being aware of the .reflectedType property is still there

OK, I will have to leave both (a) addressing this, and (b) deciding whether it's actually worth addressing this, to you, since I am not using an IDE and TypeScript is perfectly happy, and maybe it's OK for the IDE not to know about it because it's sort of an "internal" property we don't want folks mucking about with anyway. (I mean really to be on the completely safe side which should make it an unsettable property etc. -- definitely not worth doing in this prototype, and maybe not ever)

(I think we have to extend some interface with & { reflectedType: string }?):

Maybe, but be careful, the type reflection remains somewhat fragile. Adding a clause like that to some type definition somewhere could make all the types go opaque and lose us all the nice detail we're getting about dependencies and signatures... That's what happened to me when I was trying to make things work by changing Signature<OP, type> in this way.

It definitely makes to start simple with a "frozen" Dispatcher, and in the second stage make it possible to configure things. And good idea to allow creating a new dispatcher based on one or multiple existing dispatchers.

Great. I think we're good to go with #19 and then implementing the "frozen dispatcher". (Although just because it's frozen implementation-wise and type-wise, I think it still should be "lazy" and not mash up each operation from all of its implementations until it's called. In fact, I think that's necessary because of possibly nested templates -- you can't unroll Complex<Complex<Complex<...>>> to arbitrary depth off the bat. So it will still be revising behaviors internally as they get called.

> I've checked out the latest version of `main`, and can successfully run `pnpm go`. There was one typo `export type FunctionDef {`, I've changed that to `export type FunctionDef = {` to make things compile. Weird, I wonder why I didn't get any compilation error?? Thanks for fixing. > The `$reflect!([...])` at the bottom looks quite neat. We'll need a clear error message when we forget to include some function there, that's definitely going to happen. I agree, but it should also be easy to add the first time it bites one of us -- at some point we assume the reflectedType property is there, so we just check first and if it's not, issue a good message. > Let's take this as the base for the POC and work it out, and after that reflect on the exact syntax again. Roger that. > The issue with TypeScript inside an IDE not being aware of the `.reflectedType` property is still there OK, I will have to leave both (a) addressing this, and (b) deciding whether it's actually worth addressing this, to you, since I am not using an IDE and TypeScript is perfectly happy, and maybe it's OK for the IDE not to know about it because it's sort of an "internal" property we don't want folks mucking about with anyway. (I mean really to be on the completely safe side which should make it an unsettable property etc. -- definitely not worth doing in this prototype, and maybe not ever) > (I think we have to extend some interface with ` & { reflectedType: string }`?): Maybe, but be careful, the type reflection remains somewhat fragile. Adding a clause like that to some type definition somewhere could make all the types go opaque and lose us all the nice detail we're getting about dependencies and signatures... That's what happened to me when I was trying to make things work by changing `Signature<OP, type>` in this way. > It definitely makes to start simple with a "frozen" Dispatcher, and in the second stage make it possible to configure things. And good idea to allow creating a new dispatcher based on one or multiple existing dispatchers. Great. I think we're good to go with #19 and then implementing the "frozen dispatcher". (Although just because it's frozen implementation-wise and type-wise, I think it still should be "lazy" and not mash up each operation from all of its implementations until it's called. In fact, I think that's necessary because of possibly nested templates -- you can't unroll `Complex<Complex<Complex<...>>>` to arbitrary depth off the bat. So it will still be revising behaviors internally as they get called.

josdejong commented

2023-10-20 12:22:22 +00:00

Ok I'll look into the .reflectedType issue next week. And indeed it may not be an issue in practice since it's an under the hood thing, I'll keep that in mind.

The lazy approach of the dispatcher is indeed at it's core I think, it makes sense to implement that right away.

Ok I'll look into the `.reflectedType` issue next week. And indeed it may not be an issue in practice since it's an under the hood thing, I'll keep that in mind. The lazy approach of the dispatcher is indeed at it's core I think, it makes sense to implement that right away.

glen commented

2023-10-20 13:55:41 +00:00

Oh, the much more significant current difficulties are in issue #19. Please take a look when you can; further forward progress is stalled until we find some approach there. Thanks.