How to implement Futures

[0c3d7r4]

For GraalJS, there's a section on Promises in the documentation:

Value jsPromise = context.eval(ID, "Promise.resolve(42);");
Consumer<Object> javaThen = (value)
    -> System.out.println("Resolved from JavaScript: " + value);
jsPromise.invokeMember("then", javaThen);

So I've tried similar with Python, given a script

from asyncio import run,sleep
from typing import TYPE_CHECKING

if TYPE_CHECKING:
 # only for static analysis / editor
 from promise_types import Promise  # will not execute at runtime

async def read1():
    print('[🥇 py]', 'reading! test/fixture/index.js')
    r = fs.read('test/fixture/index.js')
    print("awaiting on",r)
    res = await r
    print('[🥇 py]', 'read test/fixture/index.js')
    return res

async def read2():
    print('[🥇 py]', 'reading test1/fixture/hello-world.js')
    r = fs.read('test1/fixture/hello-world.js')
    print("awaiting on",r)
    res = await r
    print('[🥇 py]', 'read test1/fixture/hello-world.js')
    return res

async def main():
    try:
        R1=await read1()
        await sleep(0.5)
        R2=await read2()
    except Exception as er:
        print('[🥇 py] ❌', str(er))
        raise

run(main())

and an adapter:

def new_future():
 loop,running=ensure_main_loop()
 fut=loop.create_future()
 return fut

In Java, we create a future:

public <T> Value newPromise(BiConsumer<Consumer<T>, Consumer<Throwable>> callback) {
  var pyBindings = context.getBindings("python");
  var new_future = pyBindings.getMember("new_future");
  var future = new_future.execute();

  // create pure Java callbacks that complete the Python future
  Consumer<T> resolve = result -> future.invokeMember("set_result", context.asValue(result));
  Consumer<Throwable> reject = error -> future.invokeMember("set_exception", error);

  // call user lambda with pure Java callbacks
  callback.accept(resolve, reject);

  return future;
 }

With the Java-based APIs:

 private Object read(Value... arguments) {
  if (arguments.length == 0) {
   throw new RuntimeException("Filename required");
  }

  var filename = arguments[0].asString();
  var encoding = arguments.length > 1 ? arguments[1].asString() : "utf8";

  if (program.DEBUG) {
   System.out.println("{🗿 graal} Starting read " + filename);
  }

  var p = program.Promise.<String>newPromise((BiConsumer<Consumer<String>, Consumer<Throwable>>) (resolve, reject) -> {
   CompletableFuture
     .supplyAsync(() -> {
      try {
       var path = Paths.get(filename);
       if (!Files.exists(path)) {
        var error = new RuntimeException("File not found: " + filename);
        throw error;
       }

       var bytes = Files.readAllBytes(path);
       var content = new String(bytes, getCharset(encoding));
       return content;
      } catch (Exception e) {
       throw new RuntimeException("Failed to read file: " + e.getMessage(), e);
      }
     }, program.virtualThreadExecutor)
     .whenComplete((content, error) -> {
      if (program.DEBUG) {
       if (error != null) {
        System.out.println("{🗿 graal} 📕 Failed read " + filename);
       } else {
        System.out.println("{🗿 graal} 📖 Completed read " + filename);
       }
      }

      final var c = content;
      final var e = error;

      program.loop(() -> {
       if (e != null) {
        var cause = (e.getCause() != null) ? e.getCause() : e;
        reject.accept(e);
        // program.error(cause.getMessage())
       } else {
        resolve.accept(c);
       }
      });
     });

   // return null;
  });

  if (program.DEBUG) {
   System.out.println("{🗿 graal} returning future");
  }

  return p;
 }

 private ProxyObject createFSObject() {
  return ProxyObject.fromMap(new java.util.HashMap<String, Object>() {
   {
    put("read", (ProxyExecutable) MyFileSystem.this::read);
    put("readBuffer", (ProxyExecutable) MyFileSystem.this::readBuffer);
   }
  });
 }

I've spent an entire weekend hacking and it just gives inconsistent results... Sometime it run, other times it doesn't... When the promise is resolved on the same tick (during debug, returning an already finished future), it's fine but it's pending first, it stops working

[🥇 py] reading! test/fixture/index.js
{🗿 graal} Starting read test/fixture/index.js
{🗿 graal} 📖 Completed read test/fixture/index.js
{🔁 loop} completed task
{🗿 graal} returning promise
awaiting on <Future finished result='// some js file'>
^ <<< fine on finished futures
[🥇 py] read test/fixture/index.js
[🥇 py] reading test1/fixture/hello-world.js
{🗿 graal} Starting read test1/fixture/hello-world.js
{🗿 graal} 📖 Completed read test1/fixture/hello-world.js
{🔁 loop} completed task
{🗿 graal} returning promise
awaiting on <Future finished result=''>
[🥇 py] read test1/fixture/hello-world.js

but when the promise is pending initially, it gets stuck

[🥇 py] reading! test/fixture/index.js
{🗿 graal} Starting read test/fixture/index.js
{🗿 graal} returning promise
awaiting on <Future pending>
{🗿 graal} 📖 Completed read test/fixture/index.js
... <<<<< does not return to where the promise was awaited

Is there more docs on futures and how i can return promised returns from my java infra?

[0c3d7r4]

The thing I understood so far, is that "It is safe to use a context instance from a single thread. It is also safe to use it with multiple threads if they do not access the context at the same time.". So basically I'm trying to make access to context through the same 1 thread throughout the program. HOWEVER when this works in JS:

 JsResult run(Source source) throws InterruptedException, ExecutionException {
  CompletableFuture<JsResult> finalResult = new CompletableFuture<>();

  // Submit everything to the JS event loop
  loop(() -> {
   try {
    var result = context.eval(source); // Eval runs on gold-event-loop thread
    var state = "pending";
    if (result.hasMember("state")) {
     state = result.invokeMember("state").asString();
    }

    if ("fulfilled".equals(state)) {
     finalResult.complete(JsResult.ok(result.invokeMember("value")));
    } else if ("rejected".equals(state)) {
     var reason = result.invokeMember("reason");
     finalResult.completeExceptionally(
       new RuntimeException(reason.toString()));
    } else if (result != null && result.hasMember("then")) {
     // Promise: attach handlers
     result.invokeMember("then", (ProxyExecutable) args -> {
      finalResult.complete(JsResult.ok(args.length > 0 ? args[0] : null));
      return null;
     }, (ProxyExecutable) args -> {
      var jsError = args.length > 0 ? args[0] : null;
      finalResult.complete(JsResult.error(jsError));
      return null;
     });
    } else {
     // Not a Promise, resolve immediately
     finalResult.complete(JsResult.ok(result));
    }
   } catch (Throwable t) {
    // Any immediate JS error
    finalResult.complete(JsResult.error(t));
   }
  });

  // Pump JS event loop until the result completes (or fails)
  eventLoop.runUntil(finalResult::isDone);

  // Stop the event loop now to avoid dangling tasks
  close();

  // Propagate exceptions to caller
  return finalResult.get();
 }

With top-level await, the result is gotten immediately as pending promise (top-level) and the event loop is freed so that other promises can get resolved, but in Python, similar code

 PyResult run(Source source) throws InterruptedException, ExecutionException {
  CompletableFuture<PyResult> finalResult = new CompletableFuture<>();
  loop(()->{
   try {
    var result = context.eval(source);
    finalResult.complete(PyResult.ok(result));
   }catch(Throwable err) {
    finalResult.complete(PyResult.error(err));
   }
  });

  eventLoop.runUntil(finalResult::isDone);

  return finalResult.get();
 }

does not even work, because context.eval on top-level code that called asyncio.run, blocks the whole of the event loop, making it impossible to resolve inner futures which are prerequisite to resolving the top-level one... it's basically a dead-lock, which does not happen in JS because context.eval returns a promise.
and even I go fancy-pancy and run the top-level python module non in the event loop but in the main program loop, those futures still won't be resolved. i don't understand how is this not documented... what is even the point of graalpython if not java integration

// event-loop related
 void loop(Runnable callable) {
  eventLoop.submit(callable);
 }
 private void loop() {
  try {
   while (running.get()) {
    var task = queue.take();
    currentTask = task;
    context.enter();
    try {
     task.run(); // the task in python doesn't get complete because it's the main loop started by us or something.
    } finally {
     context.leave();
    }
   }
  } catch (InterruptedException ignored) {
   // System.out.println("error 100 in the event loop");
  }
 }