Implementing Indexes – Replacing the Dictionary

This is the fourth entry of the series about implementing full-text search indexes in Pharo. All started with the first entry Implementing Indexes – A Simple Index, where we present the need for having indexes in large images. Then, we have the second entry is Implementing Indexes – Who uses all my memory, where we analysed the first version and the problems it has. And then, the third entry Implementing Indexes – Compressing the Trie where we show some improvements in the Trie implementation.

This fourth and final entry analyses the remaining problem in the implementation: the space taken by the identity dictionary.

Remember the Problem

When we check the result of analysing the memory impact of our solution, as we have seen in the previous blog entries. We have the following table:

Class name# InstancesMemory (Bytes)Memory(%)
Array159,6037,696,27236.46%
CTTrieNode159,6035,107,29624.20%
IdentityDictionary159,6033,830,47218.15%
Association159,6023,830,44818.15%
ByteString9,244349,9681.66%
ByteSymbol9,244294,0081.39%
CTTrie1160.00%
UndefinedObject1160.00%
SmallInteger9000.00%
Character6400.00%
Total21,108,496100.00%
Memory footprint of our solution.

We can see that the main memory is taken by 4 classes (Array, CTTrieNode, IdentityDictionary and Association). Also, it is clear that we have a relation between the number of instances of these classes and the amount of nodes in the Trie.

If we check the raw structure of our nodes we have something like this:

Inspecting our Trie

We can see that each node has an IdentityDictionary with a Character as key and a CTTrieNode. This creates the structure of nodes that we have in the Trie. From this, we can explain the number of instances of IdentityDictionary but where are all the arrays and associations? They are taking 55% of the memory, so we have to find them.

If we continue exploring, we can see how IdentityDictionary is implemented in Pharo.

Inspecting an IdentityDictionary

Dictionaries in Pharo are implemented by using an internal array. This array contains Associations. Each association has a key and a value. Those are the keys and values of the dictionary. We can see this when we continue inspecting the objects.

Also, the associations are stored in the array. The associations are not in the array in sequential positions, there are holes in the middle. Each association is stored in the position that is associated with the hash of the key. So, it is easier to look for the association when accessing by the key (remember, that the Dictionaries are optimized to access by the key). You can check the comments in classes like Dictionary and HashedCollection to understand this behavior.

Finally, to improve the speed of adding new elements, the arrays have always free positions.

This design is great to provide a fast access to the information, to speed up the access and the insertion of new elements. However, as always, we are trading speed for space. In our specific case, we want to improve the memory footprint.

Basically we need to address:

  • Remove the need to have an IdentityDictionary and an Array. Can we have a single instance?
  • Remove the use of associations
  • Remove the empty slots in the array.
  • Doing all the previous without destroying the access performance 🙂

A compromise solution

When solving problems in the real world we need to make trade-offs. Our solution is slower, but it takes less space. So, we need to balance these two problems.

We have solved the problem by implementing the dictionary using a single array without associations. The TrieNode will store each pair key-value directly in a compact array (the array is created every time with only the occupied space). Each key-value pair is stored in the array in sequential positions, and in the order they were added to the node.

For example if we have the pairs key -> value, added in that order:

$C -> nodeC.
$A -> nodeA.  
$B -> nodeB. 

The array will contain them as:

{$C. nodeC. $A. nodeA. $B. nodeB} 

So, in the optimized version the nodes have this structure:

Optimized Nodes, reimplementing the dictionary behavior

The keys are stored in the odd positions of the array, and the values in the even positions. Each key is next to its value. So, it is still possible to access the value from the key and the key from the value, but it requires to traverse the whole array.

Benefits of the solution

If we analyse the impact of the new solution, we can see that the memory footprint has been reduced to a less of a tenth of the original solution. This is the combination of the optimization of the last blog post and the one done here.

Class name# InstancesMemory (Bytes)Memory(%)
ByteString31343757,57637.16%
CTOptimizedTrieNode22102530,44826.02%
Array12859456,49622.39%
ByteSymbol9244294,00814.42%
CTOptimizedTrie1160.00%
UndefinedObject1160.00%
SmallInteger6400.00%
Total2,038,560100.00%

A clear improvement is the size of the arrays. Not only we have less arrays (because we have less nodes, thanks to the improvement done in the previous blog post) but also each array occupies less space, because they have less empty slots.

Problems of the Solution

As we have said before, there is no perfect general solution, but correct solutions for specific situations. In this case, we have put the accent in the memory impact, but it has the following drawbacks.

  • Our solution implements a custom data structure, so we needed to implement it from scratch. This introduces possible points where we have bugs and problems that we have not seen.
  • We are not taking advantage of the implementation of Dictionary in the image, so we are duplicating code.
  • We are penalizing the creation of new nodes, as the array has to be copied into a larger one.
  • The access to this data structure is linear, so we are penalizing the access time also. The complete array has to be traversed to access or check if an element is there.
  • We have to carefully crafted the code to access the data structure not to penalize the execution time more than we can allow. This code is performance sensitive and any change to it has to be carefully tested and measure; benchmarks and profiling are good friends.

These drawbacks are not important in our current situation, but can make this optimization technique unusable.

Conclusion

In this Trie-logy of blog entries we have presented a real problem that we have seen during the development of Pharo. The intention of this serie is to not only present a solution but to present the tools and the engineering process we did to arrive to the solution. The process and the tools are more valuable than this specific solution, as the solution is only valid in our context. So, we consider a good practice that can be useful to other developers in completely different situations.

Implementing Indexes – Compressing the Trie

This is the third entry of the series about implementing full-text search indexes in Pharo. The second entry is Implementing Indexes – Who uses all my memory, where we analysed the first version and the problems it has. You can start reading from this post, but there are things that we have presented in the previous one. For example, how we detected the problem with the basic implementation. Also, maybe if you are new you can check the first post: Implementing Indexes – A Simple Index.

Remembering the Problem

In the last post, we have seen that our implementation is not optimal. It was creating a lot of nodes to store the data we are keeping. In that version, we had 18.484 values stored in the trie, but for those, we needed 159.568 node instances; a ratio of 9 nodes per value. This is unacceptable.

This problem was coming for the way of storing the elements in the Trie. We are creating a node for each character in the key, even if these nodes does not include a branch in the Trie. We have the following structure:

As we have told, it is desirable to collapse all these nodes, and reducing the memory impact.

Collapsing useless Nodes

In the data structure we are analysing, a node provides crucial information in any of the following scenarios:

  • It maps a path from the root to a given value.
  • It has more than a single child, because it has an split in the key path.

So we want to reduce the previous Trie to a new one like this:

In this new implementation, the chain of intermediate nodes that do not provide crucial information is collapsed in a single node. We can see, that all the light blue nodes have been collapsed in a single green one.

For supporting this, we have changed that the node now holds a String instead of a single Character. The key path from the root is now concatenating strings.

With this improvement, we have passed from an occupation rate from 13% to 100%.

This improvement has a little trade-off: We are making the graph simpler in data impact, but we have increased the complexity in the look-up and insertion of new nodes.

This new implementation is available side-by-side with the unoptimized version.

If we create the same example of the previous post, but with the optimized trie:

behaviors := CTOptimizedTrie new. 
SystemNavigation default allBehaviorsDo: [ :aBehavior | 
       behaviors at: aBehavior name put: aBehavior name ].

To perform the analysis, we have used the same tool, SpaceAndTime, that we have used in the previous post.

stats := GraphSpaceStatistics new
rootObject: behaviors;
yourself.

stats totalSizeInBytes.
stats totalInstances.
stats statisticsPerClass.
stats statisticsPerClassCSV.

We can see the reduced number of nodes in the Trie, and its memory impact. We have passed from 159.568 instances in the previous version to 22.102 instances in this version. We passed from occupying 5.106.176 bytes to 530,448 bytes in nodes.

Additional Strings

However, this is not perfect. As we are keeping a String instead of Characters we have to create such instances in memory. In Pharo, the Strings are objects and are allocated in memory, while Characters are immediate, and they are encoded in the referencing instance. This represents that we have additional 22.099 instances, representing 407.704 additional bytes.

At a first glance it looks like a problem, but if we see that we are just having less than 1MB (adding the nodes and the Strings) against the 5MB of the previous solution. We see that there is an improvement.

This is a nice example, that even having more instances of Strings we have a clear advantage with the new solution. It teaches us that we need to measure before doing an statement.

Splitting Nodes on Insert

As we have said with this new implementation, it is required to split nodes when adding elements in the Trie. We have a more complex insert and delete operation when inserting and removing keys compared with the base implementation.

The following image presents an example.

Adding a new value to the Trie requires to create a new node, and also it might require to split an existing node if part of the key is in a single node.

In this case, the green node, should be split in two, to handle the insertion of the pair ‘ac’ -> 3.

At first glance, it looks that new implementation is slower than the old. But… to assert anything we have to measure it.

We can measure the time to generate the whole Trie of behaviors. For doing so, we will use the #timeToRun message of blocks. So, we execute to measure the times of the new implementation

[
   behaviors := nil.
   behaviors := CTOptimizedTrie new. 
   SystemNavigation default allBehaviorsDo: [ :aBehavior |
   behaviors at: aBehavior name put: aBehavior name ]
   ] timeToRun. 

And to measure the base implementation:

[
  behaviors := nil.
  behaviors := CTTrie new.
  SystemNavigation default allBehaviorsDo: [ :aBehavior |
  behaviors at: aBehavior name put: aBehavior name ]
  ] timeToRun.

From the results we can see that the new implementation takes 169 ms and the old implementation 210 ms. As we can see, the initial considerations was misleading.

Again, we have to measure. If we don’t measure it is impossible to compare different solutions.

Conclusion

This post presents a technique that we have used to improve the Trie implementation. Although, the most important part of this post is that we show how to measure the qualities of a solution. Also, we have shown that without measuring, it is impossible to compare or even to take good decisions.

Using previous experience to evaluate a solution is important, but it can be misleading. Measuring similar problems shows us the responses.

We still have an entry in this series. In the last entry, we are going to present how we solved the problem with the IdentitySet and how we have finally went to a tenth of the memory consumption.

Transcript: the misunderstood global

In this blog post, I will discuss why using Transcript can be really badly used. I will present that with some simple care we can develop modular solutions that are flexible and can take advantages of using Transcript without the inconvenients. 

As a general remark, if you want to log better use a real system logging PLAIN real objects and not just dead strings. Because you can do a lot more with objects than mere strings. You can use Beacon (whose core is available in Pharo by default)http://www.humane-assessment.com/blog/beacon or other logging frameworks as the one developed by Cyril Ferlicot and based on dynamic variables https://github.com/jecisc/TinyLogger.

Now let us imagine that you still want to log strings.

Transcript: A misunderstood object

Transcript is a kind of stdout on which you can write some strings outputs. It is cheap. The class exposes a stream-based API (and this is a really important design point as we will see in the future).

Here is a typical bad use of Transcript

myMethod
    Transcript show: 'foo' ; cr  

It is bad because it hard codes a reference to transcript while Pharo proposes some helpers methods such as traceCr:

myMethod
   self traceCr: 'foo'

Some developers may think that this is not important but it can help you if one day you want to control the logging and for example use an object with the same API but to do something else. So avoid hardcoding globals. But there is more. 

The real concern

The problem amongst others is that Transcript is a singleton and in fact an UGLY global variable. Once you use it for real in your code, you basically killed the modularity of your program and the only thing that you can do is to hope that nothing bad can happen.

Let us look at a concrete simple case. The microdown Parser came (yes we removed this) with a simple method named closeMe:

MicAbstractBlock >> closeMe
	
    Transcript << 'Closing ' << self class name; cr; endEntry

So this method is producing a little trace so that the parser developer could understand what was happening. So you can think that this is ok.

There are two main problems:

  • First what if you want to deploy your application in a system where you do not want at all to get Transcript its class and all its family. I’m thinking for example about people producing minimal images.
  • Second, when Pharo is built on Jenkins all the tests are executed because we love tests. And this Transcript expression produces dirt on the build log. You do not want to have read such trace when you are trying to understand why the build is not working.
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicCodeBlock
Closing MicHeaderBlock
Closing MicHeaderBlock
Closing MicHeaderBlock
Closing MicHeaderBlock
Closing MicHeaderBlock
Closing MicHeaderBlock
Closing MicHeaderBlock
Closing MicListItemBlock
Closing MicListItemBlock
Closing MicOrderedListBlock

You can say that I’m exaggerating. But let us see the good way to have a log and be able to unplug it.

Encapsulation to the rescue

The solution is really simple. Just use object-oriented programming and encapsulation. To support the Parser developer, we can simply add a stream to the class.

For example we define a new variable to and initialize it to a write stream.

MicAbstractBlock >>initialize
   super initialize.
   logStream := WriteStream on: (String new: 1000)

Then we can rewrite the method closeMe as follows

MicAbstractBlock >>closeMe
logStream << 'Closing ' << self class name; cr

Then we can provide a simple setter method so that the developer can set for example the Transcript as a stream to write to. 

MicAbstractBlock >>logStream:  aStream 
logStream := aStream

If we do not control the creation of instances of the class using the stream, we will have difficulties to configure it. So if want to be able to configure the class to use a different logger, we can define a class variable so that we can send message to the class and at initialization time, we can take the value from the class variable instead of hardcoding the WriteStream.

The net result is that we have the control and we can decide what is happening. In addition we can write tests to make sure that the logging is correct. Because using Transcript makes this a brittle exercise since someone else may write to the Transcript when you do not expect it. 

Conclusion

Transcript is not bad per se. It promotes bad coding practices. Developers should stop listening to the sirens of easy and cheap global variables. With a little bit of care and a limited infrastructure is to possible to get the best of both worlds: modular objects and taking advantages of the existing infrastructure whose Transcript belongs to. 

Object-centric breakpoints: a tutorial

The new Pharo debugger is shipped with object-centric breakpoints. An object-centric breakpoint is a breakpoint that applies to one specific object, instead of being active for all instances of that object’s class. We have two kinds of object-centric breakpoints:

  • The haltOnCall breakpoint: give it a method selector and a target object, and the system will halt whenever the target object executes the corresponding method.
  • The haltOnAccess breakpoint: for a target object, the system will stop whenever the state of that target object is read or written. It is possible to specify which variable whose accesses will trigger the breakpoint.

Installation

If you use the object-centric experimental image, you can skip this part and start the tutorial. Else, follow installation instructions here https://github.com/StevenCostiou/Pharo-Object-Centric-Tutorial.

The tutorial

Context: the OCDBox objects loop

In this example, we will use a class named OCDBox. It models a trivial object named box that holds objects. It has two instance variables, elements and name, and a few methods in its API. In particular, the method addElement: adds an object into the box.

We will use a test to practice object-centric breakpoints. In this test, we instantiate 100 boxes. We iterate over all these boxes to:

  • add an object to each box,
  • print the box on the Transcript.

When you execute the test, the Transcript shows you each box that is printed in the iteration loop. If you look at the code of the OCDBox class, you will see that:

  • adding an element to a box modifies its name,
  • the printing method uses the name to display the box in the Transcript.

Object-centric breakpoints

In the following, we demonstrate the use-cases of the haltOnCall and haltOnAccess breakpoints. Each time, we start by demonstrating the breakpoint through a video, then we explain the use-case and how to identify situations where that object-centric breakpoint might help. All videos have sound.

Warming up with object-centric breakpoints

The following video shows how to install object-centric breakpoints through the inspector. Basically, we create two box objects b1 and b2, and we install various object-centric breakpoints on b2.

Installing object-centric breakpoints
  • We start by inspecting b2,
  • We select the name instance variable in the inspector, and install an object-centric breakpoint through the contextual menu:
    • a halt on read will stop the execution each time the name variable is accessed,
    • a halt on write will stop the execution each time the name variable is written to,
    • a halt on access will stop on both read and writes of name,
  • We select one of the methods of b2 in the inspector, and install a halt on call through the contextual menu:
    • each time this method is called, and the receiver is b2, then the execution halts

In our example, we install a halt on read breakpoint on the name instance variable of b2, and a halt on call breakpoint on the name: method of b2. For each case, the execution will halt only for methods reading the name instance variable in b2, or when b2 calls its name: method. The execution will not halt for b1, nor for any other object.

In the following, we apply our breakpoints on a debugging example. Instead of using the inspector, we install object-centric breakpoints directly from the debugger, as we would do in a real debugging session.

Removing object-centric breakpoints

Object-centric breakpoints are garbage collected along with objects: if your object disappears, then so does the breakpoint.

For existing objects, removing an object-centric breakpoint is as simple as inspecting the object (for example in the debugger), going into the breakpoint pane, then selecting the breakpoint and using the context menu to remove it.

Stopping when a specific object receives a particular message

The test iterates over a hundred boxes objects, and to each object it sends the addElement: message. In this exercise, we select one box object among all the boxes the test is iterating, and we install an object-centric breakpoint on the addElement: method of that object. Then, we proceed the test and the addElement: method is called on each of the boxes. The execution only stops when the selected box executes the addElement: method.

Getting to the object of interest

The first step to use an object-centric breakpoint is to get to the object you want to debug.

For now, we do that by putting a first breakpoint into our code. When the execution stops, we navigate in the debugger to find objects of interest and install object-centric breakpoints.

In the following, we put a self halt. in the test, before iterating over the boxes objects. We execute the test and start from there.

Breaking when the box of interest recives the #addElement: message
  1. After adding a halt (see above), we execute the test:
    • The execution halts and opens a debugger,
    • we are able to select an object to debug it.
  2. We select one of the boxes in the boxes collection:
    • Double-click on the boxes variable in the bottom inspector,
    • choose a box in the items pane by double-clicking on the box.
  3. In the opening inspector, go into the meta-side and select the addElement: method
  4. Right-click on that method, and select halt on call in the menu:
    • The breakpoint is installed on this method, and scoped to our box object,
    • you can see the breakpoint in the breakpoint pane (if you look at other boxes, there are no breakpoint).
  5. Proceed the execution:
    • The test iterates over the boxes and sends the addElement: message to each box,
    • only the box that you instrumented breaks on this method.
Use-cases for the halt on call breakpoint

The typical use-case for this object-centric breakpoint is when you have many instances of the same class running in your program, while you are interested to debug a method for one specific object among these instances. You are interested to answer the following question: “When is this particular object executing this particular method during the execution?”

Our box example illustrates this case. You need to debug a target method for a specific instance of OCDBox, while the test iterates over a hundred of them. Putting a breakpoint in the OCDBox class will stop the execution for every instance calling the target method.

Now imagine that you want to debug a display method in a graphical object. You will have much more instances sharing that display method. Using conventional breakpoints is tedious because it will stop every time one of the graphical object uses that display method.

In contrast, you use the halt on call breakpoint to debug a specific method for a specific object because:

  • You want to avoid designing super-complex conditional breakpoints to filter your object — sometimes you don’t even have enough information to discriminate your object,
  • you do not know when (or if!) the object you want to debug will call the method, so you may not know where to insert a standard breakpoint in the code — and there might be a lot of call sites to that method so you do not know which one your object will go through,
  • you want to avoid stepping many times in the debugger before getting to the point where your object calls the method to debug — also you do not want to miss that point by extra-stepping by mistake!

Breaking when the state of a specific object is written to

In this exercise, we reuse our test iterating over a hundred box objects. We select again a box among all the iterated boxes, but this time we want to stop when the name instance variable of that box is written to. To that end, we install an object-centric breakpoint on all write accesses to that variable in the selected object. Then, we proceed the test and the execution only breaks when the name instance variable of the selected box is modified.

Breaking when the name instance variable of our box object is written

As before, we execute the test and a debugger opens before the boxes are iterated in a loop. In this loop, each box is being printed one by one through the crTrace method.

We select one of the boxes in the boxes collection:

  • Double-click on the boxes variable in the bottom inspector,
  • choose a box in the items pane by double-clicking on the box.

In the opening inspector, go into the raw view and select the name instance variable. Right-click on that method, and select halt on write in the menu:

  • The breakpoint is installed on all write accesses to this variable, and scoped to our box object,
  • you can see the write accesses and the breakpoint installed on them in the breakpoint pane.

We proceed the execution and it breaks when the name variable of the selected box is modified. There is no halt when other boxes have their name variable modified.

Use-cases for the halt on access breakpoint

The typical use-case for this object-centric breakpoint is when you have many instances of the same class running in your program, while you are interested to know when the state of one specific object is modified. You are interested to answer the following question: “When is the state of this particular object modified during the execution?”

Our box example illustrates this case. Putting a breakpoint on every access to the name variable is long, tedious and error-prone (you may forget some). Watchpoints, Variable Breakpoints or Data breakpoints may help: those tools are able to stop execution when you access an instance variable defined in a class. However, they will stop the execution each time the name variable is modified in any instance of the OCDBox class (here, at each loop iteration!).

Imagine again that you are debugging a graphical object. You are interested to know when a given property of that object (i.e., an instance variable) is modified. You do not want all the graphical objects sharing a property to break then execution when that property is modified.

Instead, you use the halt on access breakpoint to stop the execution of a program when a particular property of a specific object is modified:

  • You want to avoid designing super-complex conditional breakpoints to filter your object — sometimes you don’t even have enough information to discriminate your object,
  • you do not know when (or if!) the property will be modified, so you may not know where to insert a standard breakpoint in the code — there might be a lot of methods modifying that property, and many call sites to those methods so you do not know which one your object will go through,
  • you want to avoid stepping many times in the debugger before getting to the point where your object calls the method to debug — and you cannot guarantee that the point where you stopped in the execution will actually modify the variable!

Conclusion: Using object-centric breakpoints for real

We have seen and practiced two kinds of object-centric breakpoints, the halt on call and the halt on access breakpoints. We used a simple example to illustrate how to use those breakpoints. However, in reality it is much more complex to decide when and how to use such breakpoints. Let’s review a few advices:

  • First, investigate to know what you have in hands,
  • put a first breakpoint at a strategic place to find your object, or inspect directly the object if you have it,
  • then discriminate: what is the information you need to find?
    • do you need to know why an object seems to not execute a method properly? Use a halt on call breakpoint on this method!
    • do you need to know when or how an instance variable is modified in an object? Use a halt on write breakpoint on this variable!

Object-centric breakpoints are not a magickal tool and, often, you may not find directly your problem. The execution may stop a few (or many!) times in the same place before you understand your problem. Still object-centric breakpoints are a faster, easier way than conventional breakpoints to find the information you need from your execution.

There are also risks: similarly to conventional breakpoints, an object-centric breakpoint will halt the execution as many times as it is hit at run time. If you install any breakpoint in code that is executed very often, you may interrupt your program for good. Think that if you put a break point on the code editor keystroke behavior, you may not be able to write code anymore!

As a closing note, other breakpoints do exist: breakpoints that stop the execution each time a particular object receives any message, or when two specific objects interact together. These breakpoints are not implemented yet, but they are scheduled for the near future!

Testing UFFI Binding with Travis

Part of software system development is to write tests. More than writing tests, developers want to set up a CI that will automatically (each commit, day, …) check the status of the project tests. While it is easy to use Travis as a CI system thanks to the work of smalltalkCI, setting up Travis to test FFI binding is more tedious.

In the following, I will present the configuration used to test Pharo-LibVLC. Pharo-LibVLC allows one to script VLC from Pharo, so one can play and control music or video.

Travis Set up

First of all, we need to set up the Travis CI. To do so, we create two configuration files: .smalltalk.ston and .travis.yml. Below, I present the default .smalltalk.ston for the VLC Pharo package. It allows one to execute the tests loaded by the VLC baseline. However, such tests might need external library to work, for instance the VLC library.

SmalltalkCISpec {
  #loading : [
    SCIMetacelloLoadSpec {
      #baseline : 'VLC',
      #directory : 'src'
    }
  ]
}

.smalltalk.ston

Installing an external library

A project using FFI  may need to install the corresponding library to work. Often, as a FFI binding project developer, you know the main library (in the case of VLC, it is libvlc). However, this library has dependencies (in the case of VLC, it is libvlccore). There are many ways to determine all the libraries you need on your system to use your FFI binding. A simple one is to use the command ldd on the main library.

Once the needed libraries are identified, we have to configure Travis to install those libraries and add them to the PATH. For the installation, the best way is to rely on the apt addon of Travis.

language: smalltalk
dist: bionic

os:
- linux

before_install:
  - . ci/before_install.sh
  - export PATH=$PATH:/usr/lib

addons:
  apt:
    update: true
    packages:
      - libvlc-dev
      - libvlccore-dev
      - vlc


smalltalk:
  - Pharo64-8.0
  - Pharo64-9.0

matrix:
  fast_finish: true
  allow_failures:
    - smalltalk: Pharo64-9.0

.travis.yml

Extra configuration (export display)

Additionally to the libraries, an FFI project may need to access to a screen. It is the case of VLC that can spawn windows to display video. Travis comes with a solution to test GUI. We then only need to export the Display to port :99. For instance, with VLC, we add a script in the before_install step of Travis. It will execute the file ci/before_install.sh that set up the display.

#!/bin/bash
set -ev

# Setup display
export DISPLAY=:99.0

before_install.sh

Use external resources with GitBridge

Finally, you may need external resources to test your FFI binding. For instance, in VLC, I need music and video files. I propose to add the needed resources in the github repository and to use them for the test. However, it is difficult to set up tests that work on your local filesystem, others filesystem, and with Travis.

Hopefully,  Cyril Ferlicot has developed GitBridge. This project allows one to access easily resources present in a local git repository. After adding GitBridge to the VLC FFI project, we can get a FileReference to the folder res of the VLC FFI project by executing the method res of the bridge

However, a last configuration set up is needed to use GitBridge with Travis. It consists on executing a Pharo script. To do so, we change the .smalltalk.ston to execute a specific file.

SmalltalkCISpec {
  #loading : [
    SCIMetacelloLoadSpec {
      #baseline : 'VLC',
      #directory : 'src'
    }
  ],
  #preTesting : SCICustomScript {
    #path : 'res/ci/addRepoToIceberg.st'
  }
}

.smalltalk.ston

Finally, the addRepoToIceberg.st file contains the script that will add to Iceberg the git project.

(IceRepositoryCreator new 
    location: '.' asFileReference;
    subdirectory: 'src';
    createRepository) register

addRepoToIceberg.st

And now you are done!

About Blocks: Variables and Blocks

This is the second post of the block closure series. This blog post is extracted from “Deep Into Pharo” and a new book under writing that will revisit chosen pieces of Pharo. We will start to play with Pharo to illustrate block and their interplay with variables.

Variables and blocks

A block can have its own temporary variables. Such variables are initialized during each block execution and are local to the block. We will see later how such variables are kept. Now the question we want to make clear is what is happening when a block refers to other (non-local) variables. A block will close over the external variables it uses. It means that even if the block is executed later in an environment that does not lexically contain the variables used by a block, the block will still have access to the variables during its execution. Later, we will present how local variables are implemented and stored using contexts.

In Pharo, private variables (such as self, instance variables, method temporaries and arguments) are lexically scoped: an expression in a method can access to the variables visible from that method, but the same expression put in another method or class cannot access the same variables because they are not in the scope of the expression (i.e., visible from the expression).

At runtime, the variables that a block can access, are bound (get a value associated to them) in ”the context” in which the block that contains them is ”defined”, rather than the context in which the block is executed. It means that a block, when executed somewhere else can access variables that were in its scope (visible to the block) when the block was ”created”. Traditionally, the context in which a block is defined is named the ”block home context”.

The block home context represents a particular point of execution (since this is a program execution that created the block in the first place), therefore this notion of block home context is represented by an object that represents program execution: a context object in Pharo. In essence, a context (called stack frame or activation record in other languages) represents information about the current execution step such as the context from which the current one is executed, the next byte code to be executed, and temporary variable values. A context is a Pharo execution stack element. This is important and we will come back later to this concept.

A block is created inside a context (an object that represents a point in the execution).

Some little experiments

Let’s experiment a bit to understand how variables are bound in a block.

Define a class named  BExp  (for BlockExperiment) as a subclass of  TestCase  so that we can define quickly some tests to make sure that our results are always correct.

TestCase subclass: #BExp

instanceVariableNames: ”

classVariableNames: ”

package: ‘BlockExperiment’

 Experiment 1: Variable lookup.

A variable is looked up in the block  definition  context.

We define two methods: one that defines a variable  t  and sets it to 42 and a block  [ t ]  and one that defines a new variable with the same name and executes a block defined elsewhere.

BExp >> setVariableAndDefineBlock

| t |

t := 42.

^ self executeBlock: [ t ]

BExp >> executeBlock: aBlock

| t |

t := 33.

^ aBlock value

BExp new setVariableAndDefineBlock

>>> 42

The following test passes. Executing the  BExp new setVariableAndDefineBlock  expression returns 42 .

BExp >> testSetVariableAndDefineBlock

self assert: self setVariableAndDefineBlock equals: 42

The value of the temporary variable  t  defined in the  setVariableAndDefineBlock  method is the one used rather than the one defined inside the method  executeBlock:  even if the block is executed during the execution of this method.

The variable  t  is  looked up in the context of the block creation (context created during the execution of the method  setVariableAndDefineBlock  and not in the context of the block evaluation (method  executeBlock: ).

Let’s look at it in detail. The following figure shows the execution of the expression  BExp new setVariableAndDefineBlock .

variable.png
  • During the execution of method  setVariableAndDefineBlock , a variable  t  is defined and it is assigned 42. Then a block is created and this block refers to the method activation context – which holds temporary variables.
  • The method  executeBlock:  defines its own local variable  t  with the same name than the one in the block. This is not this variable, however, that is used when the block is executed. While executing the method  executeBlock:  the block is executed, during the execution of the expression  t  the non-local variable  t  is looked up in the  home context  of the block i.e., the method context that  created  the block and not the context of the currently executed method.

Non-local variables are looked up in the  home context  of the block (i.e., the method context that  created  the block) and not the context executing the block.

 Experiment 2: Changing a variable value

Let’s continue our experiments. The method  setVariableAndChangingVariableBlock  shows that a non-local variable value can be changed during the evaluation of a block.

Executing  BExp new setVariableAndChangingVariableBlock  returns 2008, since 2008 is the last value of the variable  t .

BExp >> setVariableAndChangingVariableBlock

| t |

t := 42.

^ self executeBlock: [ t := 2008. t ]

The test verifies this behavior.

BExp >> testSetVariableAndChangingVariableBlock

self assert: self setVariableAndChangingVariableBlock equals: 2008

Experiment 3: Accessing a shared non-local variable.

Different blocks can share a non-local variable and they can modify the value of this variable at different moments.

To see this, let us define a new method  accessingSharedVariables  as follows:

BExp >> accessingSharedVariables

| t |

^ String streamContents: [ :st |

t := 42.

self executeBlock: [ st print: t. st cr. t := 99. st print: t. st cr ].

self executeBlock: [ st print: t. st cr. t := 66. st print: t. st cr ].

self executeBlock: [ st print: t. st cr. ] ]

The following test shows the results:

BExp >> testAccessingSharedVariables

self assert: self accessingSharedVariables equals: ’42

99

99

66

66

BExp new accessingSharedVariables  will print 42, 99, 99, 66 and 66.

Here the two blocks  [ st print: t. st cr. t := 99. st print: t. st cr  ]  and  [ st print: t. st cr. t := 66. st print: t. st cr ]  access the same variable  t  and can modify it. During the first execution of the method  executeBlock:  its current value  42  is printed, then the value is changed and printed. A similar situation occurs with the second call. This example shows that blocks share the location where variables are stored and also that a block does not copy the value of a captured variable. It just refers to the location of the variables and several blocks can refer to the same location.

Experiment 4: Variable lookup is done at execution time.

The following example shows that the value of the variable is looked up at runtime and not copied during the block creation. First add the instance variable  block  to the class  BExp .

Object subclass: #BExp

instanceVariableNames: ‘block’

classVariableNames:

package: ‘BlockExperiment’

Here the initial value of the variable  t  is 42. The block is created and stored into the instance variable  block but the value to  t  is changed to 69 before the block is executed. And this is the last value (69) that is effectively printed because it is looked up at execution-time in the home context of the block. Executing  BExp new variableLookupIsDoneAtExecution  return  ’69’  as confirmed by the tests.

BExp >> variableLookupIsDoneAtExecution

^ String streamContents: [ :st |

| t |

t := 42.

block := [ st print: t ].

t := 69.

self executeBlock: block ]

BExp >> testVariableLookupIsDoneAtExecution

self assert: self variableLookupIsDoneAtExecution equals: ’69’

Experiment 5: For method arguments

We can naturally expect that method arguments referred by a block are bound in the context of the defining method.

Let’s illustrate this point. Define the following methods.

BExp >> testArg5: arg

^ String streamContents: [ :st |

block := [ st << arg ].

self executeBlockAndIgnoreArgument: ‘zork’]

BExp >> executeBlockAndIgnoreArgument: arg

^ block value

BExp >> testArg5

self assert: (self testArg5: ‘foo’) equals: ‘foo’

Executing  BExp new testArg: ‘foo’  returns  ‘foo’  even if in the method  executeBlockAndIgnoreArgument:  the temporary  arg  is redefined.

The block execution looks for the value of  arg  is its definition context which is the one where  testArg5:  ‘  arg  is bound to ‘foo’ due to the execution of method  testArg5 .

 Experiment 6: self binding

Now we may wonder if  self  is also captured in a block. It should be but let us test it.

To test we need another class.

Let’s simply define a new class and a couple of methods.

Add the instance variable  x  to the class  BExp  and define the  initialize  method as follows:

Object subclass: #BExp

instanceVariableNames: ‘block x’

classVariableNames:

package: ‘BlockExperiment’

BExp >> initialize

x := 123.

Define another class named  BExp2 .

Object subclass: #BExp2

instanceVariableNames: ‘x’

classVariableNames:  ”

package: ‘BlockExperiment’

BExp2 >> initialize

x := 69.

BExp2 >> executeBlockInAnotherInstance6: aBlock

^ aBlock value

Then define the methods that will invoke methods defined in  BExp2 .

BExp >> selfIsCapturedToo

^ String streamContents: [ :st |

self executeBlockInAnotherInstance6: [ st print: self ; print: x ] ]

BExp >> executeBlockInAnotherInstance6: aBlock

^ NBexp2 new executeBlockInAnotherInstance6: aBlock

Finally we write a test showing that this is indeed the instance of  BExp  that is bound to  self  even if it is executed in  BExp2.

BExp >> testSelfIsCapturedToo

self assert: self selfIsCapturedToo equals: ‘NBexp>>#testSelfIsCapturedToo123’

Now when we execute  BExp new selfIsCapturedToo , we get  NBexp>>#testSelfIsCapturedToo  printed showing that a block captures  self  too, since an instance of  BExp2  executed the block but the printed object ( self ) is the original  BExp  instance that was accessible at the block creation time. The printing is  NBexp>>#testSelfIsCapturedToo  because  BExp  is  subclass of  TestCase .

Conclusion

We show that blocks capture variables that are reached from the  context in which the block was defined  and not where there are executed. Blocks keep references to variable locations that can be shared between multiple blocks. In subsequent posts we will show how this is optimized by the Pharo compiler and VM to avoid expensive lookups.

[Pharo features] Advanced run-time reflection

In this series of blog-posts, we are explaining in more detail the Pharo features described in https://pharo.org/features.

The source code describes programs statically, but this perspective is insufficient if you want to fully understand the behavior of the computer system. The source code can, for example, describe a complex graph of data objects, although its concrete shape depends on input data and state of the application. To examine the real-time status of the system, you need to have tools that allow you to see the state of objects and relations between them. It is especially true if the program does not behave in an expected way, and you need to debug it. For modern development environments, to provide such tools for run-time analysis is an essential feature.

In Pharo, you can manipulate your program as any other objects. In fact programs are objects too. So to provide tools to examine your running program, Pharo needs to be able to inspect itself too. For this reason, this feature is named “reflection“. Pharo is a computer system that can examine and modify itself.

Pharo is truly strict in that and, on the level of the object memory, does not hide anything against itself. At the same time, it exposes all this information and capabilities to the programmers. So programmers can look inside every living object in the object memory, examine its state defined by instance variables, modify them and write their own tools to do the same. Of course, it does not limit this feature only on your objects: all the tools such as the browser are using reflection to introspect objects that are representing program itself (and Pharo itself). 

Inspectors

The standard Pharo tools for examination of objects are named “inspectors“. An inspector exposes the internal structure of objects and, usually, you use multiple of them at the same time, because, in most cases, you want to see more than one object at once. It allows multiple inspectors to be opened on the same object too and each one can show a different aspect of the object. The inspectors use so-called pagers for diving into the internal structure of the objects – to inspect objects that are referenced by instance variables of the examined object. We will talk about inspectors more deeply in a separate article of this series.

An Inspector inspecting itself.

 

Surprisingly, Pharo allows the opposite inspecting direction too. It is easy to know the objects that some object is referring but to know the objects that are referencing some object is hard, especially if this it should be fast. It is an extremely useful feature, mainly in cases when you are trying to find memory leaks.

Usually, a memory leak is a name for the state when you allocate a piece of memory explicitly and accidentally never free again while nobody has a pointer to it so it cannot be reused. In extreme but not uncommon case, it can cause exhaustion of available memory with all the nasty consequences. For managing the memory, Pharo uses automatic garbage collector, so it is safe of this kind of memory leaks, but it does not mean that it is safe from wasting memory. The garbage collector cleans only the objects that are not referenced by any other object, so if an object keeps a reference while it should not, it will not be garbage collected.

For detection and solution of such cases, Pharo offers several tools such as ReferenceFinder that finds the shortest path of references between two objects. If the starting point is a root object of the garbage collector, you can detect why your object is not garbage-collected as expected.

You first need to know that you have some memory leak before you try to solve it. Pharo has the next reflective feature that enables it. It allows enumeration of living instances of a given class. So in a situation where you expect that a class should not have any instance, you can easily prove it. We will talk about this feature in more details later too.

Practical demonstration

The short video we created for the demonstration of Pharo reflection, uses these techniques.

  • First, we ask the class SystemWindow to return some instance of it using a message with the unsurprising name:someInstance. We open an inspector on the resulting object sending such instance the message inspect.
    • SystemWindow someInstance inspect
  • Then we look at the internals of this object. We can see, for example, the rectangle that forms boundaries of the window
  • We send a message pointersTo to this window. It returns a collection of all objects that have a direct reference to the window object and open another inspector on it. This collection has 73 items.
  • We choose one of them. A graphical object (morph) that is part of the window and is responsible for resizing of the window by mouse in one direction.
  • Then we look at its bounds, the Inspector dives to show an object that represents it. It is a rectangle that has two points named origin and corner. We click on the second one to see the internals of it. The point has two instance variables with coordinates x and y containing numbers.

 

One basic rule of object-oriented programming declares that objects must be encapsulated. No-one has the rights to look at their internal structure or modify them. What we have demonstrated clearly violates this rule! Right?

Actually not. If the Inspector looks at an object private content or it modifies the instance variables directly, it does it by sending special messages to the object. These messages are instVarNamed:, instVarNamed:put: and so on. The object itself can still determine how it will react on such messages or if it will implement them at all. It can, for example, return false information or pretend that the state of the object was changed while it was not.

It is just extremely convenient to implement such methods for all the objects correctly, so the programmer has a right and complete view on the state of the system so he can study, understand and change it quickly.

For programmers, it is great to have the system fully exposed, but what about users? It may be dangerous to give them the power to examine and change everything they want, and the approach of Pharo may look like a recipe for disaster. But, in reality, Pharo isn’t different from any other computer system. You just do not let the users or attackers execute any code they create.

Ed: your new Emergency Debugger

“What is the nature of your debugging emergency?”

—Ed.

In this post, we’ll speak about debugging emergencies due to system error handling failures. We’ll cover the current tools available to revert code changes provoking such failures and introduce Ed, the new emergency debugger with improved revert capabilities.

What is a debugging emergency?

Sometimes when you execute code, you get a strange failure that opens that window and blocks your execution. This is the emergency evaluator invite.

It says a lot of things and it is scary, and it is common to just close it as fast as possible to get back to our Pharo image and forget about it.

Unfortunately, this means that there was a system error handling failure, it may happen again, and sometimes it totally breaks your image. This is a debugging emergency!

A debugging emergency is a system error handling failure

A system error handling failure happens when the debugger cannot be opened after an exception is raised. Most of the time, this happens because an error occurs within the debugger itself, which prevents it to work properly. Consequently, the debugger cannot open to debug its own error. This is an error-handling failure.

Because the system failed to handle the error, the whole system crashes. The system falls back to the emergency evaluator invite (see above) and waits for the user to decide what to do. We are in a debugging emergency state because tools are broken (here, the debugger) but nevertheless we need to recover from that error. At this point, the only tool still able to work is the emergency evaluator.

The only tool able to deal with an error-handling failure is the emergency evaluator

The emergency evaluator is a rudimental tool with a super limited user interface. It is started from the emergency evaluator invite. It answers to two commands only:

  • revert: reverts the last modified method to the last known version of that method (in this screenshot, it just failed doing that… why?).
  • exit: quits the emergency evaluator, dropping the original error.

Any other command is treated as Pharo source code: it is compiled and executed.

We need better emergency handling because current tools are limited

Emergency tools are too limited: they do not provide enough information, that information is not explorable and on top of that, they are buggy!

The first limitation is the restricted information that is given by the emergency evaluator invite. It is a printed representation of the last 20 elements of the call stack. All other debugging information is dumped, to only keep that printed stack. Because it is only printed, a lot of contextual information is lost. It is often hard to see in that stack where it crashed. The problem is that users have to rely on that information to devise what to do in the emergency evaluator!

Then comes the emergency evaluator. Its revert command is interesting. It allows us to recover from a fatal method change that provokes the error. But there are many problems. Among others, it is buggy and does not work all the time. Then, it only reverts the last modified method in the system. What if the problematic method is not the last modified one, but another one modified much before?

When reverting does not work, the only option left is to manually recompile the buggy method. This is tedious and error-prone: we have to remember the original class name, the method’s source code and rewrite it without any error. Although this is possible for simple methods, it does not scale. Finally, in case of error, the emergency evaluator will not give us any feedback and we’re left in the dark.

ED: the Emergency Debugger

Ed is an enhanced emergency evaluator that keeps as much debugging features as possible before falling back to the emergency evaluator. Ed provides a structured and navigable call stack from which users decide their debugging actions.

Ed is designed as a minimal, last-resort debugger to handle error handling failures. The goal of Ed is to provide a debugging tool with the less possible system dependencies to ensure that it can work even if almost everything else breaks.

Ed always opens on your domain code failure: it does not show you the debugger error. Your concern is to debug your code. However, it is possible to see a stack trace of the debugger failure, and to debug it with Ed.

Ed’s improved emergency stack view

Ed presents a navigable stack instead of a string representation of the stack. Instead of dropping the debug session when a system error handling failure occurs, Ed keeps it and uses it to provide us with contextual information about the failure. The stack that Ed presents is a stack of execution contexts extracted from the debug session.

The navigable stack helps us understand the error and where it originally happened:

  • We can see and navigate the call stack like in a standard debugger (top pane)
  • We can see the argument value (if any) printed between parentheses near each method in the call stack
  • We can see the source code of the method selected in the stack (middle pane), and see exactly what expression failed or is being executed (in red)

Ed’s method version selector

For a selected method in the call stack, Ed provides access to the different versions of that method. The source code of each version is displayable and navigable. Viewing methods’ versions helps in deciding which one is the method to revert. In addition, it does not matter if the right method was the last modified method in the system. It’s highly probable that this method is somewhere in the call stack, and we can navigate the call stack!

  • We can see and navigate the selected stack method’s versions using left/right arrows: bottom pane shows version 2/15 of the selected method
  • We can revert the method to the selected version by using the revert command
  • Once a method is reverted, the retry command allows you to retry opening a real debugger on your error, and the proceed command will attempt continue execution from there.
  • Any method in the stack can be selected and reverted to a previous version

 

What else can Ed do?

Ed provides a prompt with basic commands. For now, Ed uses a simple debugging API with minimal commands. Remember: you got here because the debugger failed while opening on an error from your domain code. So Ed just gives you another chance to debug your error. Maybe you are interested in debugging the debugger failure, in which case Ed allows you to switch context to debug that failure. It is also possible to retry the opening of the standard debugger after reverting a method or to terminate all non-vital system processes to kill potential image-breaker running code.

  • Ed provides some help to know all available commands: type "h"
  • The set of commands is provided by the debugging API of Ed
  • Using the showDebuggerError command, Ed shows the debugger failure that led you here, and the debugDebugger command opens a new Ed on the debugger failure.

Can Ed fail?

Yes, it can. Any program can. In case Ed fails, the system fallsback to the original emergency evaluator. For now, a failure in Ed should not block you. For instance, imagine that you get a debugger failure while debugging an exception in your domain code. First, Ed will open on your exception (not the debugger failure) so that you can debug your code. Second, if an exception is raised from within Ed and that it breaks Ed, the emergency evaluator will open on your exception again. Debugging the debugger is not going to be on your soulders, and you can focus on your bugs!

Note that, in the future, this last part is something we are going to improve to bring more flexibility to developers. Settings will enable the debugging of debugger failures or disable them to try to always find a debugger that can debug your exceptions.

Additionaly, the default emergency debugger can now be selected in the settings. You can choose if you wish to experiment Ed or to stay with the old emergency evaluator:

Ed is a prototype, but it moves fast!

Ed is still a prototype, and it needs better testing and experimentation. For instance, it still has system dependencies that can be removed, it has code that can be improved.

In the end, the goal is also to give users a read-eval-print-loop prompt with a strong debugging API. For now, Ed only has it’s minimal emergency API. However, it has been designed in a way that its API is extensible by additional tools, improving Ed’s potential capabilities. In the future, we want to explore remote debugging with Ed. For instance, Ed could act as a minimal debugging infrastructure that you can remotely connect to through an simple http connection. And if your debugging needs grow, Ed could receive a bytecode upload containing new debugging tools and API. We will experiment and explore this in future articles about debugging IOT devices.

About Blocks: Basics 101

This blog post is a first and short of a series where we will explore block-closures. This post will cover basic elements while the other will start to go over more semantical aspects using simple examples. This blog post is extracted from “Deep Into Pharo” and a new book under writing that will revisit chosen pieces of Pharo.

Lexically-scoped block closures, blocks in short, are a powerful and essential feature of Pharo. Without them it would be difficult to have such a small and compact syntax.
The use of blocks is key to get conditionals and loops as library messages and not hardcoded in the language syntax. This is why we can say that blocks work extremely well with the message passing syntax of Pharo.

What is a block?

A block is a lambda expression that captures (or closes over) its environment at creation-time. We will see later what it means exactly. For now, imagine a block as an anonymous function or method. A block is a piece of code whose execution is frozen and can be kicked in using messages. Blocks are defined by square brackets.

If you execute and print the result of the following code, you will not get 3, but a block.
Indeed, you did not ask for the block value, but just for the block itself, and you got it.

[ 1 + 2 ]
>>> [ 1 + 2 ]

A block is executed by sending the message value to it.
More precisely, blocks can be executed using value (when no argument is mandatory), value: (when the block requires one argument), value:value: (for two arguments), value:value:value: (for three) and valueWithArguments: anArray (for more arguments).

These messages are the basic and historical API for block execution.

[ 1 + 2 ] value
3
[ :y | y + 2 ] value: 5
7

Some handy extensions

Beyond the value messages, Pharo includes some handy messages
such as cull: and friends to support the execution of blocks even
in the presence of more values than necessary.
cull: will raise an error if the receiver requires more arguments than provided.

The valueWithPossibleArgs: message is similar to cull: but takes
an array of parameters to pass to a block as argument.
If the block requires more arguments than provided, valueWithPossibleArgs:
will fill them with nil .

[ 1 + 2 ] cull: 5
>>> 3
[ 1 + 2 ] cull: 5 cull: 6
>>> 3
[ :a | 2 + a ] cull: 5
>>> 7
[ :a | 2 + a ] cull: 5 cull: 3
>>> 7
[ :a :b | 1 + a + b ] cull: 5 cull: 2
>>> 8
[ :a :b | 1 + a + b ] cull: 5
>>> error because the block needs 2 arguments.
[ :a :b | 1 + a + b ] valueWithPossibleArgs: #(5)
>>> error because 'y' is nil and '+' does not accept nil as a parameter.

Other messages

Some messages are useful to profile execution:

  • bench. Return how many times the receiver block can be executed in 5 seconds.
  •  durationToRun. Answer the duration (instance of class Duration ) taken to execute the receiver block.
  • timeToRun. Answer the number of milliseconds taken to execute this block.

Some messages are related to error handling – we will cover them when we will present exceptions.

  • ensure: terminationBlock. Execute the termination block after evaluating the receiver, regardless of whether the receiver’s evaluation completes.
  •  ifCurtailed: onErrorBlock. Execute the receiver, and, if the evaluation does not complete, execute the error block. If evaluation of the receiver finishes normally, the error block is not executed.
  • on: exception do: catchBlock. Execute the receiver. If an exception exception is raised, execute the catch block.
  • on: exception fork: catchBlock. Execute the receiver. If an exception exception is raised, fork a new process, which will handle the error. The original process will continue running as if the receiver evaluation finished and answered nil, an expression like: [ self error: ‘some error’] on: Error fork: [:ex | 123 ] will always answer nil to the original process. The context stack, starting from the context which sent this message to the receiver and up to the top of the stack will be transferred to the forked process, with the catch block on top. Eventually, the catch block will be executed in the forked process.

Some messages are related to process scheduling. We list the most important ones. Since this Chapter is not about concurrent programming in Pharo, we will not go deep into them but you can read the book on Concurrent Programming in Pharo available at http://books.pharo.org.

  • fork. Create and schedule a Process evaluating the receiver.
  • forkAt: aPriority. Create and schedule a Process evaluating the receiver at the given priority. Answer the newly created process.
  • newProcess. Answer a Process evaluating the receiver. The process is not scheduled.

In the next post we will present some little experiences to show how blocks captures variables.

On the edge of class rules

Pharo has one simple basic rule: everything is an object. But the objects themselves are not entities living in an abstract universe and do not drink the dew of lilies of the valley for breakfast. They exist in the object memory served by the virtual machine. The virtual machine defines an interface that strictly specifies how the objects have to look like and what rules they need to follow.

For objects, the amount of rules is very low – basically just one. Pharo is a class-based object system that requires every object to have an assigned class. Every object is an instance of some class. 

Classic Class rules

For classes, the amount of rules is much wider. Classes are the source of the behaviour of objects, and they shape how the objects will look physically in memory. For this reason, every class needs to be an object with at least three instance variables. The placement and order of these variables are strictly given. These variables are:

  • superclass – can be nil for the class hierarchy roots
  • methodDict – a dictionary of method name symbols and associated compiled methods. It really needs to be a dictionary, and because this data structure is so essential for the virtual machine, the internal structure of it is strictly defined as well
  • format – an integer that encodes the kinds and numbers of variables

These three instance variables are defined in the class named Behavior that is the basic class of all classes in Pharo. If you try to add a new instance variable that will change the order of these three, the virtual machine crashes because it changes the expected layout of the classes in memory. Even if you try to define a superclass of Behavior and add a new instance variable there, it has the same fatal result because it modifies the memory layout as well. 

Classes themselves are objects too, so they need to follow the same rules. Most of the classes have a name and are nicely placed in the Pharo class hierarchy. They are mentioned in the system dictionary (a namespace) and the system browser can work with them. But not all classes need to be like that. Pharo provides an option to create so-called anonymous classes that behave more like standard objects. You can inspect them, but the other development tools do not know about them. They have various usages – mostly in special and strange cases – but they may be handy especially during testing because such classes are not registered in the system, do not need to have a unique name and are very easy to discard. They are created using the messagenewAnonymousSubclass that you send to an existing class.

aClass := Object newAnonymousSubclass.

We said that objects are instances of classes and classes are objects. If you think about such rules while drinking something uplifting, you may ask yourself: Is it possible to have an object that is an instance of itself? Can an object be its own class? Does Pharo allow such an edge case? And if yes, is it any useful?

Funky Classes Objects

How to try it? The most straightforward way how to do that would be to create an object (e.g., as an instance of the class Object) and then assign its class to itself. But here you quickly hit two problems. First, as we said, the virtual machine has some restrictions on the memory layout of classes that you need to follow. The second problem is that Pharo objects have no direct way of how to change their class to another. They do not understand any message like class: that would do this job.

Surprisingly, both of these problems are quite easily solvable! Let’s see how.

First, we will create an anonymous class. Unlike most of the anonymous classes, we will create classes that have Class as a superclass (an anonymous metaclass :)). That means that such anonymous class will define the same instance variables as Class. The three instance variables that make the virtual machine happy plus some other variables that make the tools happy.

aClass := Class newAnonymousSubclass.

Such class looks pretty standard in the Inspector.

Then we create an instance of such a class. 

anObject := aClass basicNew.

Inspecting of this class does not offer a very satisfying view. All instance variables are referencing the default value – nil. Moreover, the print string is broken.

There is an easy fix of this shattering state. We will just copy all instance variables from the original anonymous class.

aClass allInstVarNames do: [ :instVarName |
  anObject instVarNamed: instVarName put: (aClass instVarNamed: instVarName) ].

 Now the object looks the same way as the anonymous class in the Inspector, so there is no need to put the picture here again. They look the same way, but they are not the same objects. And, we still haven’t reached our goal to create an object that is its own class.

(anObject == aClass)
>>> false. (anObject class == anObject)
>>> false.

But we are close. Now, the object is an instance of our anonymous class and the object itself is already quite a suitable class.

(anObject class == aClass) 
>>> true.

So we may do the same step that Pharo does when it migrates object to a new class. For example, when you change the structure of the original class object. We may let our anonymous class become the desired object. 

aClass becomeForward: anObject.

That updates even the low-level pointer in the object memory that defines a class of our object. It was pointing to the anonymous class that becomes our object. The original anonymous class will stop to exist; we do not need it anymore.

(anObject class == anObject) 
>>> true.

The mission accomplished! We have created an object that is its own class.

Now, we can start to play with it. It is a class so we can add methods to it, which can be useful for creating mocks. 

anObject compile: 'ultimateAnswer ^ 42'.

And the object will understand it. As the only object in the object memory.

anObject ultimateAnswer 
>>> 42

So what?

Great, but is it useful? It can be. This technique is not something new. I have seen it in the Prototypes package by Russell Allen from from year 2005 that allowed prototype-based approach in the class based system. His original implementation cannot work in Pharo; however, the principle stays. Like JavaScript and unlike Self, it allows only a single target of delegation – here named superclass

It is more an interesting oddity than something that you would use every day in your programs. But it can imagine it may be useful mocking, in some debugging scenarios or modeling programs. Last but not least, it is a nice and amusing feature that demonstrates the flexibility of the object model of Pharo.

Here is the entire code:

aClass := Class newAnonymousSubclass.
anObject := aClass basicNew.
(aClass allInstVarNames) do: [ :instVarName |
    anObject instVarNamed: instVarName put: (aClass instVarNamed: instVarName) ].
aClass becomeForward: anObject.

Conclusion

We got some fun around classes, and we hope that you enjoyed it. If you want to understand more conceptually objects, classes and metaclasses you should read the little book: ‘A simple reflective object kernel’ http://books.pharo.org/booklet-ReflectiveCore/ . It is not about Pharo, but it shows in Pharo how to define a minimal reflective kernel, and you can learn a lot of fun aspects of OOP (like instantiation, allocation, lookup,…) in the process.