FFI interface

[hiperiondev]

Describe the feature you’d like

I've been reviewing the code a bit and it seems to be a very good language to embed in microcontrollers.
Does Toy have any FFI interface?

[Ratstail91]

Hi @hiperiondev!

Yes, it has native function support i.e. functions written in C.

//declared in toy_literal.h
typedef int (*Toy_NativeFn)(struct Toy_Interpreter* interpreter, struct Toy_LiteralArray* arguments);
typedef int (*Toy_HookFn)(struct Toy_Interpreter* interpreter, struct Toy_Literal identifier, struct Toy_Literal alias);

//declared in toy_interpreter.h
TOY_API bool Toy_injectNativeFn(Toy_Interpreter* interpreter, const char* name, Toy_NativeFn func);
TOY_API bool Toy_injectNativeHook(Toy_Interpreter* interpreter, const char* name, Toy_HookFn hook);

These allow you to define native functions which are simply "injected" into the interpreter's current scope - alternatively, a hook is invoked with the Toy keyword import, like so:

import standard;

print clock(); //clock() provided by standard

A few libraries are made available this way within the repl folder; they start with the filename "lib_".

This is the intended way to expand the language's functionality - but I haven't written many libraries beyond what is currently provided (help would be greatly appreciated ;) )

Is there anything else you need help with?

[hiperiondev]

Thanks for your quick response!
As soon as I have a bit of time I'm going to make an implementation for ESP32 and I'll test it.
I'll let you know when it's ready

[Ratstail91]

Thank you so much! The full docs are on the website https://toylang.com, if you need to look up exactly why something exists the way it does.

And of course, I'm always available for any questions you have.

[hiperiondev]

First attempt to embedd on ESP32: https://github.com/hiperiondev/esp32_toy

I have made some changes

• Separate libraries: Are no part of interpreter. Better to handle them independently (could also be a separate project).
• Reorder project: source and include files in separate folders.
• Replace pragma once by guard: pragma is not well supported everywhere and can cause problems in some systems (For example: in the ESP-IDF plugin for Eclipse does not allow indexing system headers)
• Some style changes: It is not really important but it is very useful to improve readability
  • System headers on top
  • Spaces, not tabs: The length of the tabs is not something standard and in different editors it may not be represented in the same way
  • repl_main is separated: it is optional to embed the system but it can be useful for some tests, but must be ported

An FTP server and a custom little filesystem are added to improve usability on a microcontroller. It would be useful to modify the file accesses with macros (fopen, fclose, etc) to facilitate the use of native filesystems on microcontrollers.

Other possible changes:

• Add a better license: I understand the philosophy of not using a license but something very permissive would improve development control (maybe MIT?)
• Add comments: Functions should be commented out in all headers to improve the possibility of development by third parties (like me :-) ). It would be ideal to use a format like Doxygen to automate documentation

That would be all for now. I'm going to finish the functionality test (for now only compile) and I'll keep you informed.

[hiperiondev]

Added test_interpreter and test_compiler. Pass all test without errors!
It would be desirable to separate the compiler and the interpreter. On devices with such low memory it is not always a good idea to compile inside the device.

[Ratstail91]

(I won't get into spaces vs. tabs - only madness lies there.)

I get the pragma change, but was separating headers & source just a preference?

License - the code is already under the zlib license, which is basically exactly the same as the MIT license, just less famous.

I've documented what each function does on the docs website, but I suppose it's a pain to go look it up.

You're in luck - the interpreter operates completely independently of the compiler, and vice-versa. Once the compiler collates the bytecode, the bytecode can be saved to a file and executed separately.

I highly recommend reading the docs site (you can skip the quickstart section for now) - https://toylang.com/ - it'll explain a lot of my thoughts about what Toy, and might help a lot.

[hiperiondev]

(I won't get into spaces vs. tabs - only madness lies there.)

Spaces vs. tabs: Bloody wars have started for much less!

I get the pragma change, but was separating headers & source just a preference?

Virtually all changes are personal preference. In the case of headers, it is convenient if they are self-documenting. It is a single place to look for references.

License - the code is already under the zlib license, which is basically exactly the same as the MIT license, just less famous.

That's good, I hadn't noticed because there is no reference in any file as a comment in the header. It would be nice to add it in each file

I've documented what each function does on the docs website, but I suppose it's a pain to go look it up.

Not only does it result in the inconvenience of needing an external reference, but using Doxygen-style comments you can automate the process of creating external documentation and the code is left with its own documentation.

You're in luck - the interpreter operates completely independently of the compiler, and vice-versa. Once the compiler collates the bytecode, the bytecode can be saved to a file and executed separately.

That's very good. I will check it. Perhaps it would be convenient to separate them into two directories to make explicit the choice of each component in the implementation

I highly recommend reading the docs site (you can skip the quickstart section for now) - https://toylang.com/ - it'll explain a lot of my thoughts about what Toy, and might help a lot.

I'll do that.

Thank you so much

[hiperiondev]

I have separated the compiler and interpreter code. It is more comfortable to compile only the interpreter in an embedded system.

Now everything seems to work correctly. I will try to do something useful to test.

[Ratstail91]

I didn't think to separate the compiler from the interpreter, partly because they both rely one the same literal/array/dictionary structures - but you do you.

If and when I make updates, are you planning on importing the updates to your branch? Or staying at 1.1.6? Because there are still small niggling corner cases that need fixing eventually (the issue tracker is full of them lol).

I'll look into doxygen - I suppose it's just another personal preference to have a separate site. I made the site super easy to update by storing it in a separate branch within the main repo - so it's basically sitting next to the source it documents.

There's a lot of assumed preferences in my code that I wasn't aware of - I suppose it's the result of my inexperience working with other coders. You never stop learning I guess.

I might convert this issue into a discussion soon, since that would make more sense to me.

[hiperiondev]

Here are my final modifications in structure: https://github.com/hiperiondev/esp32_toy/tree/main/components/toy_lang/Toy

I didn't think to separate the compiler from the interpreter, partly because they both rely one the same literal/array/dictionary structures - but you do you.

I find the Toy language very interesting for embedded applications due to its similarity with C and simplicity of implementation, it is an excellent option as a glue language. But on a microcontroller resources are scarce and precious (ESP32 has 200k of RAM, with FreeRTOS OS plus a few things to work hopefully 80/90k are free for the program!)
In this case separate the compiler from the interpreter is very important. Every byte is important.

If and when I make updates, are you planning on importing the updates to your branch? Or staying at 1.1.6? Because there are still small niggling corner cases that need fixing eventually (the issue tracker is full of them lol).

At first I can follow "by hand" the changes that are made. If you think my changes are correct we could try to integrate this way. Your project, your decision.

I'll look into doxygen - I suppose it's just another personal preference to have a separate site. I made the site super easy to update by storing it in a separate branch within the main repo - so it's basically sitting next to the source it documents.

Doxygen not only allows you to automate, with a very clear syntax, the documentation, but also integrates very well with other systems such as Sphinx (widely used by https://readthedocs.org/ for example). If you want to involve more people in development, it is a very useful tool.
I have added the skeletons of the comments in the headers.

There's a lot of assumed preferences in my code that I wasn't aware of - I suppose it's the result of my inexperience working with other coders. You never stop learning I guess.

No problem. The decisions depend a lot on whether you intend to allow and organize the participation of more people in the development or follow it only as a personal project.
As I already said:Your project, your decision.

I might convert this issue into a discussion soon, since that would make more sense to me.

I agree with that.

[Ratstail91]

Here are my final modifications in structure

I think dictionary should be in common, and builtin should be in the interpreter folder - otherwise, cool layout.

At first I can follow "by hand" the changes that are made.

That's probably the best approach, since the directory layout changed so much. I'll keep mine how it is for now (because Box and Skylands depend on it, and I'm not in the mood to tweak those too), but it's possible I'll follow suit at some point.

I have added the skeletons of the comments in the headers.

Cool, thanks - I'll have a peek soon.

The decisions depend a lot on whether you intend to allow and organize the participation of more people in the development or follow it only as a personal project.

It started out as only a side-project, something to reduce stress, but it's kind of taken off - people really seem to like it, and others have already contributed in the past with things like building on MacOS and error testing (though the error testers were a group of people who... let's not get into that here).

I'd love more people to be involved, and it honestly makes me really happy what you're doing.

[Ratstail91]

Converted the issue to a discussion.

Edit: Also cleaned up the discussions in general.

[hiperiondev]

Here are my final modifications in structure

I think dictionary should be in common, and builtin should be in the interpreter folder - otherwise, cool layout.

The idea of the separation is to be able to compile only the interpreter without the compiler.
To separate the files I have searched the functions in the workspace to find the references in other files.
The toy_literal_dictionary functions are not used by the compiler. Is there a reason why it should be in common?

At first I can follow "by hand" the changes that are made.

That's probably the best approach, since the directory layout changed so much. I'll keep mine how it is for now (because Box and Skylands depend on it, and I'm not in the mood to tweak those too), but it's possible I'll follow suit at some point.

No problem. For now the modifications are more of style than functionality.
So I'm going to clean up and detach the project (makefiles completely unusable) to decouple it from the implementation.
If, in the future you change your file structure to follow this model I can easily modify my project

I have added the skeletons of the comments in the headers.

Cool, thanks - I'll have a peek soon.

The decisions depend a lot on whether you intend to allow and organize the participation of more people in the development or follow it only as a personal project.

It started out as only a side-project, something to reduce stress, but it's kind of taken off - people really seem to like it, and others have already contributed in the past with things like building on MacOS and error testing (though the error testers were a group of people who... let's not get into that here).

I'd love more people to be involved, and it honestly makes me really happy what you're doing.

Thank you so much. You have really done an excellent job!

For the topic of using Toy in embedded systems, a deeper documentation on the operation of the interpreter and its internal components would be very useful. Probably some modification should be made to improve memory usage.
A good example of how this can be done is the Lua VM documentation.

[hiperiondev]

New project: https://github.com/hiperiondev/uC-TOY

[hiperiondev]

I have finished the skeleton of doxygen documentation.
You can see the result in: http://raw.githack.com/hiperiondev/uC-TOY/main/documentation/doxygen/html/index.html

[hiperiondev]

I have added all the documentation available. There are many undocumented things

[Ratstail91]

Is there a reason why [dictionaries] should be in common?

It's more a feeling of "this is one of the languages building blocks" - it's the same feeling I get from arrays and literals.

Thank you so much. You have really done an excellent job!

That means so much to me.

For the topic of using Toy in embedded systems, a deeper documentation on the operation of the interpreter and its internal components would be very useful.

I'll see what I can do during my pass to add comments.

New project: https://github.com/hiperiondev/uC-TOY

Ohh interesting. So you're going to tweak it specifically for microcontrollers?

I have finished the skeleton of doxygen documentation.

Cool - I actually want to talk to you about that in #84.

There are many undocumented things

I'll need to resolve that going forward - the documentation that is there is intended largely to aid people in using Toy, rather than building on top of it. I guess I'll need to change the approach there.

[hiperiondev]

Is there a reason why [dictionaries] should be in common?

It's more a feeling of "this is one of the languages building blocks" - it's the same feeling I get from arrays and literals.

Thank you so much. You have really done an excellent job!

That means so much to me.

For the topic of using Toy in embedded systems, a deeper documentation on the operation of the interpreter and its internal components would be very useful.

I'll see what I can do during my pass to add comments.

New project: https://github.com/hiperiondev/uC-TOY

Ohh interesting. So you're going to tweak it specifically for microcontrollers?

It is not my real intention but a microcontroller requires particular optimizations due to its limited resources and direct handling of the hardware. Some elements must be minimal or static (for example it is not a good idea to have 64bit data handling directly in the interpreter, it is also not a good practice to use int for everything, intx_t or uintx_t is advisable). It's not clear to me what else they're using Toy for, but you've told me that other projects use it and, for that reason, you can't change its structure.

I have finished the skeleton of doxygen documentation.

Cool - I actually want to talk to you about that in #84.

I will follow this topic in that place.

There are many undocumented things

I'll need to resolve that going forward - the documentation that is there is intended largely to aid people in using Toy, rather than building on top of it. I guess I'll need to change the approach there.

I think that this is mandatory to adapt it in a more generic way. For this reason I had advised you to see the internal documentation of LUA, since it was created specifically for this (in fact, it was originally made to work inside television in Brazil and send programs through the TV Ginga standard)

https://www.lua.org/notes/
and , in particular:
https://www.lua.org/doc/jucs05.pdf
https://the-ravi-programming-language.readthedocs.io/en/latest/lua_bytecode_reference.html

[hiperiondev]

NOTE: In Issues only have Bug Report and Feature Request. A category for Questions about language use would be useful.

[Ratstail91]

intx_t or uintx_t

This is on the TODO list, actually marked as a much-needed element for the next major version.

It's not clear to me what else they're using Toy for

It's actually embedded into a game engine - that was always it's initial purpose: to allow easy modding of video games.

The engine is here: https://github.com/Ratstail91/Box
The game is here: https://github.com/Ratstail91/Airport

They're still in early development, so forgive how bare-bones it is.

lua

Oh wow, that's quite the read. I'll start digging into this soon.

A category for Questions about language use would be useful.

No worries, will add that now.

[Ratstail91]

@hiperiondev I should emphasize that Toy's development is done in a way that tries to reduce stress on me, as I'm suffering from some mental health issues. So I might not be able to work as fast as is needed, but I'll do my best to provide what help you need.

For now, I might not get too much done until the weekend.

I also have another major project called Egg Trainer which, while in an ok state, also needs my attention from time to time.

I felt like I should let you know this stuff to pre-empt any issues with how fast things get done (and to remind myself to take it easy).

[hiperiondev]

@hiperiondev I should emphasize that Toy's development is done in a way that tries to reduce stress on me, as I'm suffering from some mental health issues. So I might not be able to work as fast as is needed, but I'll do my best to provide what help you need.

Don't worry about this, health always comes first!
Take your time and don't follow my rush and emotions.
Each one at his speed, no one is obliged to anything here

For now, I might not get too much done until the weekend.

I also have another major project called Egg Trainer which, while in an ok state, also needs my attention from time to time.

I felt like I should let you know this stuff to pre-empt any issues with how fast things get done (and to remind myself to take it easy).

I do my project in my free time (because my work is quite a lot!).
The recommendations on the documentation I make due to my experience in this field. It is preferable to spend a little more time on this and save a lot more later. Nobody likes documents, neither do I, so it's better to do it before and not at the end :-)

[Ratstail91]

Don't worry about this, health always comes first!

Thanks for understanding.

The recommendations on the documentation I make due to my experience in this field. It is preferable to spend a little more time on this and save a lot more later. Nobody likes documents, neither do I, so it's better to do it before and not at the end :-)

I totally agree :)

[Ratstail91]

@hiperiondev Before I forget

Toy_findLiteralIndex was renamed to Toy_private_findLiteralIndex.

A quick find and replace should fix that.

I've also bumped the version number, but you can leave updating your stuff until I actually do a release.

[hiperiondev]

Changed in my code

[hiperiondev]

I think this will be interesting for you:

https://mega.nz/folder/UzR0RLIJ#XBuYNEV5Fs6DJlcyCu5asA

[Ratstail91]

Sweet, thank you! I'll have a read tonight or tomorrow.

Right now I need to work on the game that is using Toy, so I'm not letting my audio guy down...

[hiperiondev]

This week I will finish integrating the minimum of HAL: fs, wifi, and GPIO

[Ratstail91]

Awesome - if you need any help, just ask.

[Ratstail91]

https://toylang.com/ - docs updated for clarity.

[hiperiondev]

Good!
I have added a minimal functionality for wifi access (connect, disconnect and scan).
It's not the most beautiful way to implement it but I'm learning how to do it. I'll probably change it, but it works now

[Ratstail91]

It's all a learning process - Toy had 5 iterations before the current branch lol.