Add support for WebSocket

[stefanak-michal]

After our calls I was thinking about building javascript library and I've finally found some time this week. My goal was to make it work in browser and also in backend (nodejs). But I hit a roadblock because TCP socket connection is not possible from browser javascript. They only way is to use WebSocket. Would you consider adding support for WebSocket into ThingsDB?

I know HTTP API is available but it doesn't work both ways so listening is not possible with it.

[joente]

I think that is a great idea. The library https://libwebsockets.org seems to be the way to go. They also support libuv, the event loop used by ThingsDB.

[joente]

@stefanak-michal , in branch https://github.com/thingsdb/ThingsDB/tree/lws I'm testing with WebSockets.
Note that it's not perfect yet, certainly not production ready.

However, it is testable.

I've added two simple files for testing:

• Python: https://github.com/thingsdb/ThingsDB/blob/lws/itest/test_ws.py
• HTML/JavaScript: https://github.com/thingsdb/ThingsDB/blob/lws/itest/ws/index.html

(My browser didn't like self signed certificates so I've added the Python file for TLS/SSL testing)

[joente]

Start with WebSocket support using something like this:

THINGSDB_WS_PORT=7681 \
THINGSDB_WS_CERT_FILE=target/localhost-100y.cert \
THINGSDB_WS_KEY_FILE=target/localhost-100y.key \
thingsdb

Or, without TLS/SSL:

THINGSDB_WS_PORT=7681 \
thingsdb

[stefanak-michal]

That sounds great. I'll look into it soon.

[stefanak-michal]

I've noticed you pack ID as unsigned int. Documentation say only 16bit, nothing about unsigned. When I tried to verify and send id higher than 32767 it didn't work. It is correct?

[joente]

For the ID it doesn't matter if you use signed or unsigned, as long as you pack the ID as 16bit. Unsigned, this number can be up to 65535 (higher than 32767). You just get back what you put in, the ID is used to map the result to the request. (responses might not be returned in the order by which the requests have been sent).

[joente]

I've improved the docs to make understanding socket connection IDs easier: https://docs.thingsdb.io/v1/connect/socket/

[stefanak-michal]

I have made Dockerfile to have ThingsDB instance with websocket support. It is available here: https://github.com/stefanak-michal/thingsdb.js/blob/dev/ci/thingsdb.Dockerfile

I just have one question. When I tried COPY (to copy executable) it didn't work. I tried all combinations for path which came to my mind. In the end I used RUN cp. Any idea?

[joente]

I think COPY is trying to copy the file from you local file system. RUN cp is fine, or just start thingsdb from the Release path. I've also changed the Docker files in the lws branch (but not tested yet if they actually work).

[stefanak-michal]

oh I didn't know ..thanks

[joente]

@stefanak-michal , I've switched the build system to CMake (https://cmake.org/) in the lws branch. Since we require libwebsockets compiled with the libuv event loop, relying on the OS-provided version was unreliable and CMake made it easier to include libwebsockets as a static build in the project. Additionally, CMake improves maintainability for future collaboration with more developers on ThingsDB.

The Dockerfiles have been updated, for a manual build you can run ./debug-build.sh or ./release-build.sh (Debug versus Release build).

If you have experience with CMake, I'd appreciate any feedback on how the build could be further improved.

[stefanak-michal]

My experience with CMake are almost none 😄

[joente]

Did you already start with a JavaScript/Node.js client?

Just some thoughts about the project:

• I think the project should be build using TypeScript?
• Maybe add the option to perform "raw" queries (There might be scenarios where you simply need the raw data)?
• Room support can be implemented similar to other connectors, where we register "rooms" to the connector. A "room" should have at least the option to define an async "onJoin" handler, which must be called after making the initial connection and a re-connect (rooms must be joinen again after a re-connect) to ensure users have a point where they can sync the state with ThingsDB.

[stefanak-michal]

yes I did. https://github.com/stefanak-michal/thingsdb.js/tree/dev

• it is in Typescript.
• what do you mean with raw?
• I'm making it as wrapper for specification. So far I'm not adding custom system.

[joente]

Nice 👍

By "raw" i meant without decoding the message. But maybe this is not relevant for the JS client. With Go and C# I found this useful for scenarios where we just needed the raw messagePack data. (Otherwise the client would unpack/decode and the next step would be to encode the message again.) But, maybe not that relevant for the JS client so ignore it.

I think the approach to build a wrapper is fine. With that foundation it will be easy to implement rooms.

[joente]

The Python client can now connect using both Socket and the new WebSockets. This is also convenient for testing.

await client.connect('ws://localhost:9270')  # Or wss:// when Secure WebSockets is enabled

[joente]

@stefanak-michal , I think we're almost ready for making a ThingsDB release with WebSocket support (v1.6.0). I've just fixed some minor issues related to Cmake on MacOS and this too seems to work fine now.

[stefanak-michal]

Sounds good to me.

JS driver works well but I'm missing the most important ingredient to finish some stuff, which is time. :)

[joente]

A start to the WebSocket documentation page is made: https://docs.thingsdb.io/v1/connect/websocket/

Once the JavaScript client is ready, we'll update the page... @stefanak-michal , would you be interested in joining the ThingsDB organization on GitHub (in that case I'll sent you an invite)?

[stefanak-michal]

Of course I'm interested.

[stefanak-michal]

Okay. I think I'm good. https://github.com/stefanak-michal/thingsdb.js

[joente]

When testing, I get an error at this line:

const view = new DataView(event.data);

Message in FireFox console:

Uncaught TypeError: DataView: expected ArrayBuffer, got Blob
    onmessage https://unpkg.com/thingsdb.js@latest/dist/thingsdb.js:1
    V https://unpkg.com/thingsdb.js@latest/dist/thingsdb.js:1
    value https://unpkg.com/thingsdb.js@latest/dist/thingsdb.js:1
    onload file:///home/joente/workspace/thingsdb/itest/ws/index.html:13
    EventHandlerNonNull* file:///home/joente/workspace/thingsdb/itest/ws/index.html:11

Do you want me to create an issue or is this familiar?

(Happened when using the browser implementation: <script src="https://unpkg.com/thingsdb.js@latest/dist/thingsdb.js"></script>)

[joente]

In my small test I used something like this to resolve that error:

new Response(message.data).arrayBuffer().then(buffer => {
   const view = new DataView(buffer);
   ...

[stefanak-michal]

Chrome complained about getting blob and wanted ArrayBuffer :D

I'll look into it.

[stefanak-michal]

fixed. I learned something new. When I do npm publish in my console, it will take what I have in dist directory and put it into unpkg. I thought unpkg will automatically execute build from the package. So I just had old version of javascript file in my dist directory.

[joente]

I've also noticed a small typo in the example:

thingsdb.query('@:stuff', 'Hello World!')

Should be:

thingsdb.query('@:stuff', '"Hello World!";')  // the `;` is not required, but the string must be wrapped in quotes

[joente]

I've added the JavaScript connector to the ThingsDB documentation: https://docs.thingsdb.io/v1/connect/javascript

[joente]

WebSockets is implemented, closing the conversation. Thanks @stefanak-michal !