Squirrel

I've recently started working on a serialization system for Squirrel that will allow the persistence of threads along with objects, similar to Lua's Pluto library or Stackless Python's pickling.  So far I've built a rough proof of concept that allows you to "pickle" suspended generators into strings and "unpickle" them at some arbitrary future point to resume execution. I'm calling the process "pickling" after the fashion of Python at the moment, but it might be cool to use a more "Squirrel-like" name sometime in the future. :-)

You can download my beta work here.

The code right now is based on Squirrel 2.2.2 stable. Here is a brief summary of the changes I have made:

• New files sqpickle.h and sqpickle.cpp containing functions for serializing generators
• base_pickle and base_unpickle are added to the base_funcs registry in sqbaselib.cpp
• A small hack has been added to the compiler in SQCompiler::Statement(), to add current stack position information to "yield" opcodes (used during pickling to avoid the serialization of "popped" stack positions)

Also some minor bugfixes that aren't in 2.2.2:

• The generator.call() bug is fixed as per this thread
• The friend VM root table reset bug(?) has been fixed as per this thread
• Added error messages for attempting to use call() on a suspended or running thread
  

To use the new features, you can just call pickle() on a suspended generator; pickle() returns a string containing the state of the generator at the moment of the call. You can "unpickle" this string back into a generator by calling unpickle(), which requires two arguments: the string to restore from, and an object that will become the environment object of the newly revived generator. The return value of unpickle() is a new generator with the same internal state as the pickled generator.

For example:

// a boring generator
function my_gen() {
    local count = 0

    while(true) {
        print("  this.count = " + this.count++)
        print(", local count = " + (count++) + "\n")
        yield
    }
}

this.count <- 0;
local a = my_gen();

print("--- original generator ---\n")
resume a
resume a
resume a
print("    >>pickled at this point<<\n")
local x = pickle(a)
resume a
resume a

// now restore the generator from the dump, with a new environment
local env = { count = 1000 }
local b = unpickle(x, env)

print("\n--- unpickled generator ---\n")
resume b
resume b
resume b

The following output is produced:

--- original generator ---
  this.count = 0, local count = 0
  this.count = 1, local count = 1
  this.count = 2, local count = 2
    >>pickled at this point<<
  this.count = 3, local count = 3
  this.count = 4, local count = 4

--- unpickled generator ---
  this.count = 1000, local count = 3
  this.count = 1001, local count = 4
  this.count = 1002, local count = 5

The unpickled generator picks up at the exact state of execution at which the original one was pickled. In this example I specified a different environment for the unpickled generator to "wake up" in, but of course you could use the same one as well.

Note that this is just a really basic demonstration, and there are some significant limitations of the system at the moment:

• The only objects you can serialize are generators. I plan to add serialization for threads and regular objects as well, but for now it's just generators.
  
• There's no system for serializing hierarchies of references, shared references, or any aggregate types, so for now any generator you pickle is only allowed to have simple types (int, float, string, null) on its stack. The exception is the environment object, which can be any object. (This is because the environment object is not serialized.)
  
• Free variables are not properly restored. If a generator's closure includes free variables, it will just crash when it tries to run after unpickling. It doesn't do anything more helpful (like give an error message), since I only just now noticed that it does this. :-)

So, any thoughts, ideas, suggestions, advice? I decided to start this because it was a feature I'd like to have for my game engine; making actor control threads part of the serialized game state should enable creating "snapshot" game saves without too much of the hassle of managing complex behavioral state. Hopefully others will find it useful as well, at least once I've advanced it beyond a trivial implementation. :-)