Reference counting as memory management has a drawback in that it is difficult to detect circular references when determining dead objects. AngelScript allows the application to register types with special behaviours to support the garbage collection for detecting circular references. These behaviours make the class a bit more complex, but you should only have to register them for a few types, e.g. generic container classes.
The difference between the garbage collected and non-garbage collected types is in the addref and release behaviours, the class constructor, and the extra support behaviours.
The GC determines when objects should be destroyed by counting the references it can follow for each object. If the GC can see all references that points to an object, it knows that the object is part of a circular reference. If all the objects involved in that circular reference have no outside references it means that they should be destroyed.
The process of determining the dead objects uses the first four of the behaviours below, while the destruction of the objects is done by forcing the release of the object's references.
Whenever a garbage collected class is created, the garbage collector must be notified of it's existence. The easiest way of doing that is to have the factory behaviour, or the class constructor call the
NotifyGarbageCollectorOfNewObject() method on the engine when initializing the class.
You may want to consider caching the typeId, so that it doesn't have to be looked up through the relatively expensive call to GetTypeIdByDecl every time an object of this type is created.
Note, if you create objects of this type from the application side, you must also notify the garbage collector of its existence, so it's a good idea to make sure all code use the same way of creating objects of this type.
For garbage collected objects it is important to make sure the AddRef and Release behaviours clear the GC flag. Otherwise it is possible that the GC incorrectly determine that the object should be destroyed.
If you plan on executing scripts from multiple threads with automatic garbage collection turned on, or if you plan on running the garbage collector manually from a background thread, then you must make sure that the object type behaviours that support the garbage collector are thread-safe. Especially the ADDREF, RELEASE, and ENUMREFS behaviours have a high probability of being invoked from multiple threads simultaneously. The RELEASEREFS behaviour will only be invoked when the Garbage Collector has determined that the object is already dead so it is guaranteed not to be invoked by multiple threads. The others, GETREFCOUNT, SETGCFLAG, and GETGCFLAG, are not sensitive as the garbage collector just use the information as a hint.
Making the ADDREF and RELEASE behaviours thread-safe is easy with the use of asAtomicInc and asAtomicDec. If the object is static container, i.e. the memory layout of the contents cannot change, then ENUMREFS is already thread-safe, but if memory layout can change, e.g. dynamic arrays or hash maps, then the iteration over the content in ENUMREFS must be protected so that it doesn't break in case the memory happen to change in the middle of the iteration.