Enigmeta

Fighting Cognitive Debt in Agentic Code with Video Overviews

2026-02-19T00:00:00Z

AI writes code faster than I can understand it — so now the system generates narrated videos of itself.

I used Claude Code to build Vamos, a polyphonic synthesizer plugin in C++20, based on Ableton's Drift. A working synth in a few hours — but since I didn't write any of the code, I'm missing the crucial part: understanding what was actually implemented.

Margaret-Anne Storey calls this "cognitive debt": gaps in human comprehension that widen even when the code itself is perfectly fine. Agentic coding amplifies this — the code arrives faster than your ability to absorb it. So I tried something different: what if the synth code also generates narrated video walkthroughs that explain it? Using Remotion for programmatic video and ElevenLabs for voice-over, the whole thing feels like a natural YouTube series.

Here's the first episode — an 8-minute walkthrough of Phase 1, from an empty project to a working eight-voice polyphonic synth:

The video pipeline

The videos are built with Remotion — each visual element (code blocks, waveform visualizers, architecture diagrams) is a React component that animates based on frame count. Because these are code, you can create domain-specific visualizations. Here's an ADSR envelope, color-coded by phase, with the overshoot line clearly marked:

The production workflow has two passes:

Pass 1: Structure and mute review. Claude Code outlines the episode, writes the narration transcript, and builds the Remotion scenes — but doesn't generate audio yet. Instead, it renders a mute video with burned-in captions. I review, give feedback, Claude Code adjusts and re-renders. This is where most of the iteration happens, without spending TTS credits.

Pass 2: Voiceover and retiming. Once the mute video looks right, Claude Code generates voiceover through ElevenLabs. The audio durations then dictate final timing: each segment's length gets converted to frames, and scene durations are recalculated to match.

A useful ElevenLabs detail: since each narration segment is a separate API call, voice quality can drift between segments. The generation script passes adjacent segment text as context (previous_text / next_text) and chains request IDs, so each segment is conditioned on the actual audio of the one before it.

The result is that code and visuals work together in ways that comments never can. When SmoothedValue code appears alongside a before/after waveform comparison — the click artifact versus the smooth ramp — you see why it matters:

Where this could go

Three levels of video overview worth exploring:

"In the weeds" — You open a PR, and a two-minute narrated video walks the reviewer through the changes.

"Diving back in" — You return from vacation, hit a button, and a video catches you up on what changed while you were away.

"Archaeology" — Historical deep dives that capture the reasoning behind decisions, the part that's hardest to recover from code alone.

Video isn't necessarily the final form — Simon Willison has been exploring web comics for the same purpose. But the pattern of narration-driven explanation (structure → transcript → mute review → TTS + retiming) works well and isn't specific to any particular type of code.

The code has become easy. Understanding is the hard part.

NodeBox: Data Visualisation Made in Belgium

2019-09-26T00:00:00Z

On 26 September 2019 I gave a talk at Dataviz BXL, a data visualisation meetup organised by Digityser in Brussels, on an evening themed "data visualisation made in Belgium". I talked about NodeBox, the tool we build at Sint Lucas Antwerpen: where it comes from, how it works, and what people have made with it. This is a cleaned-up version of that talk, with the slides on the left.

I work at Sint Lucas Antwerpen, an art college where we teach painting, sculpture, photography, graphic design, advertising and illustration. Because it is an academic institution it also does research, and that is where I work, in a research group called EMRG. We develop NodeBox.

NodeBox 1

NodeBox started from software like John Maeda's Design by Numbers (1999), made to teach programming to designers. You write very small programs that draw into a square of 100 by 100 pixels. It is an imperative way of programming: each line is a command that does something.

Processing, by Casey Reas and Ben Fry, grew out of the same ideas, and we started NodeBox around the same time.

NodeBox 1 (2004) works much like Processing. It was built on DrawBot by LettError, which is still active today. The visuals are on one side, the code on the other, and the errors at the bottom, which is where you look most of the time.

You type a command and it gets painted. This is called the painter's model, and you know it if you have worked with Canvas or OpenGL. A command sets some state, such as the fill colour, and the drawing commands after it use that state.

It works well, but you have to keep the state of the program in your head at every point. If you copy a few lines to another place and forget the fill line, the result changes.

We run a lot of workshops with this software, and none of the students are programmers. They draw basic shapes in Illustrator or elsewhere, bring them into NodeBox and start manipulating them. Organic Dots by Veronika Schmidt is an example. These are generative designs rather than data visualisations, and the only data in them is random.

Some of these programs do take input. The Evil Font by Valtteri Vitakoski is a generative typeface with one parameter, evilness. At zero it is a plain sans serif. Add some evil and the letters start sprouting thorns, eyes and swords, until at full evilness the word disappears under them.

What I like about it is that this slider is not one you will find in Illustrator. Students invent their own parameters, make a slider for them, and connect it to whatever they want. People who work in data visualisation know this idea well, but it is still good to see students discover it.

Spamghetto, by the Italian design studio TODO, takes the spam in your mail folder and turns it into wallpaper. You can send them your own spam and they run their NodeBox script on it to make a personal pattern. In the close-up you can read a mail "from the desk of the president of Trump University".

NodeBox 3

The biggest problem with NodeBox 1 turned out to be the programming. Many of the people we teach are graphic designers and illustrators who have no background in programming and are not much interested in it. We tried teaching them, and it works for some people but not for everybody.

So we tried to turn the model around, and that became NodeBox 3. It is a visual application that requires no programming. You build a visualisation from small building blocks, which we call nodes, and each node does one part of the work. (This screenshot is of NodeBox Live, the later web version, which I come back to below.)

Think of a factory assembly line. The cars move down the line and every station does one part of the job.

We are going from data to visuals instead of building cars, and there are intermediate steps on the way. We process the data until it has the right shape. At some point we turn a value into a visual property: the width of a bar in a bar chart, the radius of a circle, a position on a map after a projection.

Here is a simple network. We import a CSV file, and lookup nodes extract the columns we need. The data is rarely in the form we want, so we convert it: we change the range from millions to pixels, or simply multiply. Then we generate shapes from the data, such as text on a path, rectangles or lines, and modify them further by aligning or colouring them. A merge node combines everything into the finished visualisation.

The whole application uses this one consistent system, which makes it easy to learn.

You don't need to understand the whole process to use it. You can take an existing NodeBox file and start changing its parameters without touching its structure. All of these designs come from the same network; only the parameter values differ.

There have been several versions. NodeBox 2 worked quite differently and is no longer supported. NodeBox 3 and NodeBox Live share the same evaluation model. NodeBox 3 is an open source desktop application, and NodeBox Live runs in the browser.

Examples

Jan Aulbach made Shipwrecks of California in a data visualisation workshop we gave in The Hague. It is a timeline of the ships that wrecked off the Californian coast, each sized by its tonnage. In the early years there are many wrecks as maritime traffic grows. Later, safety improves and wrecks become rare, but when one happens it is a much bigger ship.

Katju Aro visualised the declining stock of fish in the North Sea, with a colour for each species. It is a dramatic picture: nearly every band goes down.

Fifty Years of Film Speech by Jonas Lekevicius and Juste Ziliute is one of my favourites because it brings everything together. They took the subtitles of the fifty most popular films of each year, over fifty years, and kept the fifty most frequent words. For every word they drew a sound wave that shows how often it is spoken over time.

A lot of data preparation happened before the data reached NodeBox. Not everything can be done inside it.

If you put the raw data into Excel with every word on top of each other, you get the chart on the top right. That is a bit unfair to Excel; you would use small multiples instead.

NodeBox let them build their own visualisation, a sound-wave shape, which suits a project about speech. Like the evilness slider, the sound wave is a custom node: a small network packaged as a single node. You attach data to it and the shape comes out. The last image is a close-up of the words changing over time.

Nicholas Rougeux visualises musical scores in a series called Off the Staff. Each image is a whole piece: Tchaikovsky's 1812 Overture, Vivaldi's Autumn, a Brandenburg Concerto. He uses NodeBox among other tools, and he writes about his process, which shows that NodeBox gets used outside the classroom too.

His making-of post explains every step. He opens the score in MuseScore, exports it from MIDI to MusicXML, converts the XML to CSV, and cleans it up in Google Sheets, because no data visualisation project is finished without a spreadsheet. Then it goes into NodeBox.

He numbered the parts of his network, which makes it a good illustration of the model I described. It starts by importing the CSV. The blue nodes process the data. They compute the current time from the frame number (NodeBox renders video at 30 frames per second and the data is in seconds), a colour per instrument, an angle starting at twelve o'clock, a circle size from the note's duration, and a distance from the centre from its pitch and octave.

From step 10 the nodes turn dark. Those are visual nodes, where numbers become shapes. A rect node, for example, has four input ports for x, y, width and height, and what it draws depends on what you connect to each.

There is much more on his site, c82.net, such as Literary Constellations (2017).

List matching

All these examples work on many data points, so where are the loops? NodeBox behaves like a functional programming language. There are no explicit loops; NodeBox loops over everything implicitly, using a system we call list matching.

When a node gets two lists, it pairs them element by element. Say the dots are positions and the numbers are data: the first number goes to the first dot, the second to the second, and so on.

If one list has a single element and the other has many, the single element is repeated. With other lengths, such as two shapes and ten positions, the shorter list wraps around and the shapes alternate. The output is as long as the longest list.

This works well in practice, and it makes it easy to see what every node is doing.

Visual debugging

Because every node does something concrete, you can always check where things go wrong. We use this when teaching: once students know a bit of NodeBox, we ask them to swap computers with the person next to them and break their network. Then everyone has to find the bug in their own.

In the demo I opened a network that visualises the words used across all Metallica albums. I expected lines inside the circle and saw none. The viewer shows the output of the node at the bottom, a combine node that merges everything. I clicked my way up through the network. The text around the circle was fine, but the coloured lines were missing. Further up, a lookup node that should have returned the list of albums returned nothing, because of a mistake in its settings. After fixing it, I stepped back down node by node, checked the lines, the colours and the combine, and the visualisation appeared.

In an imperative program with loops, a mistake somewhere in the middle is hard to locate. In NodeBox you go up the network one node at a time, compare the data you see with the data you expect, and find the node where they differ.

NodeBox Live

NodeBox 3 works well, but it cannot do anything interactive. About two years before the talk we made NodeBox Live, which brings the same model to the web.

This demo builds a dashboard widget from tweets about telecom operators. Here they are imported from a file, but NodeBox Live also supports real-time data streams. A scale node maps the earliest and latest dates to pixels, those become points, and the port you connect them to decides whether they drive x or y. A sentiment analysis library splits the tweets into positive and negative. We turn them into ellipses, colour them, add a legend and combine everything.

To put it on a dashboard, we make it smaller, copy a snippet of iframe code and paste it into the dashboard page. The embed stays live, so you can keep adjusting it afterwards. Because the network is a procedure, you can point it at the data of another operator. The other charts on the dashboard, such as the word cloud, are NodeBox Live networks too.

On the web you get everything the web offers, including interactivity, and I like to mix the serious work with less serious student projects. Quinten De Meester was fascinated by how slow the revolving doors in shopping centres are, and wondered what would happen if you could change their parameters.

His Raging Revolving Door (2015) lets you set the motor speed, how fast people walk, and how many of them there are, like a busy shopping season versus a normal day. Then you can turn everything up and watch.

De Standaard

Because we work in an academic context, we have to think about how NodeBox can sustain itself, and a commercial version is one option. So we worked with De Standaard on Jouw gemeente in cijfers ("your municipality in numbers"), a large dataset the Flemish government publishes about every municipality.

We knew the format in advance but not the data, so we prepared everything and had only a few days once it came out. We worried the report would be so beautiful that we would be out of a job. This is what came out: about 150 pages of tables like this one for every municipality.

We made a day in the life of someone living in each municipality, from Edegem and De Panne to Bruges and every other one, and the illustrated skyline changes with the place.

NodeBox generated everything, including the HTML and the text. It has nodes for working with text, so a sentence like "as for the 21,834 other inhabitants of Edegem, your day begins" comes straight out of the network.

Each part of the day has its own visualisations. On the commute, the number of cars on the road follows the data: 75% of people in Zoersel drive to work or school almost every day, against 43% in Antwerp. A row of people shows the share of inhabitants of foreign origin, 19.2% in Edegem.

Every segment is a NodeBox Live network, and the networks are grouped together. It was a fun project, because illustrators, people working on the data and people building the visualisations all worked on the same thing.

The future of NodeBox

At the time of the talk there were two applications, NodeBox 3 and NodeBox Live. Our plan was to bring them together, with the web version as the standard and something like Electron, which Slack, GitHub Desktop and Spotify use, to run it on the desktop.

We think it matters to have a tool that is open source, freely available and supported by its community. The speaker before me argued that these tools have to last over time, and I agree. Open source means you can keep using them for as long as you need.

You can find NodeBox at nodebox.net.

Counting Words with Python 3

2018-03-07T00:00:00Z

This post will get you set up to start processing datasets with Python 3. We will learn how to use the terminal, install Python if you don't have it already, and write a simple word-counting script that returns the result in CSV.

A big focus will be on getting you to "Google level experience", meaning that you will know the right things to search for if you want to do something not mentioned in this article, or if you get stuck. Knowing what to search for is important to tackle your own data processing jobs.

During the many NodeBox workshops I've given I noticed that students would eventually be able to figure out the visualization on their own, but they had trouble getting the data in the correct shape. Being able to acquire the data (e.g. from an external API), converting it to a format you can use is key to making good visualizations. But first, we need to talk about the terminal.

Full source code of this project is available here: https://gist.github.com/fdb/1c7234fb49757df588c1302f44f0f125

The Terminal is Your Friend

You probably already heard this a number of times, but the Terminal, also known as the command line, is an immensely powerful tool that we're going to use.

The command line has been around since the dawn of the computers. Even with all graphical interfaces, people keep coming back to it. The reason is simple: it gives you access to the raw power of your computer. If you want to do something that no one else has done, the terminal is the place to be.

A good way to look at the terminal is as having a chat conversation with your computer. When you start it up it will be waiting for your command. You give it an instruction and press enter, and the terminal will execute it. It's a silent partner: often if the command succeeds it will be silent. It doesn't ask you to brag about the fact you just created a directory to your friends on Facebook.

Fully understanding the command line can take years. Luckily we just need a few very basic commands: how to switch to the right directory, how to install some software, and how to start and stop our scripts.

First we need to find the Terminal. On a Mac, the program is called "Terminal" and can be found using Spotlight, or in the Utilities subfolder under the Applications folder. On Windows, you need to install the Bash terminal. Instructions are here.

Once you open the Terminal you will get a blank screen with some text. Again, think of it as a chat conversation, not a text editor. You can't move the cursor to a random position on the screen. Instead you just type a command, press enter, and new text will appear below.

The place where the terminal is waiting for your input is called the prompt. After you've executed a command, the terminal will show a new prompt, informing you it's ready for the next command. You don't always have to wait for the previous command to complete though. Often with long-running jobs (like downloading some images) you can just open another terminal tab or window and continue working there.

Let's try our first command. Open the terminal if it's not open already. Then, type pwd and press ENTER.

The terminal will print out your current working directory. In my case it's /Users/fdb. "fdb" is my user name, so yours will probably be somewhat different.

Already there are two important concepts we need to discuss. One, what's a "working directory"? And two, what is our working directory (in other words, where are we?)

Working Directory

Whenever we type a command in the terminal it executed in the current working directory. Compare it to a builder constructing a house. If you enter a certain room in the house and execute an operation - say, paint a wall, then that will happen in the room you're currently in. It's the same with the terminal. Instead of being in a room, you're in a certain directory on your file system.

Finding your way

Modern operating systems try to obscure the directory structure from you but it's actually quite important. Knowing where your files are is essential when working in the terminal.

When the terminal starts up it'll be in your home directory (the builder analogy is not that far fetched!). In your home directory you Will have subdirectories - rooms - that contain even more subdirectories (ehm, cupboards, in our analogy). The main subdirectories in your home directory will sound familiar: the desktop, your documents, downloads, movies, pictures.

The home directory itself is in a folder called "Users". If you share your computer with other people, they will have their home folder under a different user name in the "Users" folder.

The Users folder itself is in the root of your computer. In addition to a "Users" folder there is also the "Applications" folders. Applications are shared between all users of the computer: if you install a new app it will be available to everybody using the same computer.

We used the pwd command to print the current working directory. Hopefully it's now clear what this prints out:

/Users/fdb

The root folder contains a directory called "Users". In that directory is a folder called "fdb" (or whatever your user name is). The different directories are separated by slashes ("/").

Listing a directory

To see which files are in a directory use the ls command. Type ls and press enter. This will show you a list of files inside of the current (home) folder. Here are mine:

Desktop
Documents
Downloads
Dropbox
Experiments
Google Drive
Library
Movies
Music
Pictures
Public

You probably recognize some: your desktop, your documents, downloads and so on. All of these are inside our home folder. Note that the desktop is also a directory, just like your downloads directory.

The Finder provides a different, somewhat more user-friendly, view on the same information. If we go to the "Go" menu in the finder and choose "Home", it opens up a window with the same files. It's important to realize that what the Finder shows and what the terminal views are two views on the same information: the files stored in a directory on your hard drive.

Changing Directories

Your projects are probably saved in the Documents folder, or the desktop, or maybe the Dropbox folder. Whatever the place we need to switch to that directory before we start executing commands. We'll use the cd command, short for change directory, for that.

To get to your desktop type the following command and press enter:

cd Desktop

If we type pwd now (don't forget to press enter!) we'll see that the working directory has changed to /Users/fdb/Desktop. That means we're in the desktop directory.

Type ls to see the files on your desktop:

Screen Shot 2018-03-06 at 23.38.57.png
Screen Shot 2018-03-06 at 23.39.13.png
Screen Shot 2018-03-06 at 23.41.43.png
Screen Shot 2018-03-06 at 23.45.14.png

My listing will have very different files from yours. The Desktop starts out empty, and in my case fills up quickly as the deadline approaches!

Installing Packages with Homebrew

Installing software in the Terminal is done through Homebrew. The package we'll need right now is Python 3, which is not installed by default on the Mac.

First, let's install Homebrew. Copy and paste the following line of code in your Terminal and press enter:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

First this will ask to confirm the directories, just press enter. Then it will ask for your password and show a small key cursor. Type your password here. You will not see any letters appear, that's okay.

If all goes well we should have a working Homebrew install. Type:

brew

and press enter to see a list of options. You'll type options after the brew command, separated by a space. The most important ones are brew install and brew search.

Installing Python 3 using Homebrew

Python 3 is not installed by default on your macOS. But maybe you have it through a different installer. Try typing python3. If the command line responds with "command not found" you don't have Python installed.

If you have Python installed, it will show some text and then the Python >>> prompt. Here, type Ctrl-D (not Command) to exit Python.

To install Python 3, type:

brew install python

After that, you should be able to type python3 (and enter) to start Python in what's called "interactive mode". It will show the version number, and then the Python prompt, indicating it's listening to your commands:

>>>

At this point, you're having a chat conversation with Python. Try typing 3 + 5 (and enter) to let Python calculate something and return a response.

Press Ctrl-D (not Command) to exit Python.

The interactive mode is useful for trying out quick things. In general though, we'll run Python by executing a scripts. This is a text file with a .py extension that contains a list of commands we want Python to do, like open a file, download a web page, and so on.

Text Editors and Project Folders

To edit files we need a plain text editor. This is not Word or LibreOffice, but a program like Sublime Text, Atom or Visual Studio Code. I prefer the latter since it has an integrated terminal, which will automatically change the directory to the current project path.

Whenever we work on a data processing folder we should create a project folder. It's generally a bad idea just to create files randomly on the desktop, but especially for data processing jobs since they can produce many intermediate folders. Also, our editor works at the folder level, automatically helping us find files in the project folder.

Create a project folder on the desktop called "dataproc". You can use the Finder, or you can type the following commands:

cd
cd Desktop
mkdir dataproc

The first cd here has no directory path and conveniently changes the directory to the home directory no matter where we are.

Our First Python Script

We're going to write a small script that will load a file, count the unique words in the file and save the word count as a CSV file.

First we need to have some content. I'm going to use the American declaration of dependence since it's freely available. You can download it here:

https://gist.github.com/fdb/1c7234fb49757df588c1302f44f0f125

But wait! Since we can now use the command line, we can also download it by using curl, a program that can fetch web pages and files from the web.

Make sure we're in the "dataproc" folder (if not use cd Desktop/dataproc to navigate to it) then type the following command:

curl -O https://gist.githubusercontent.com/fdb/1c7234fb49757df588c1302f44f0f125/raw/bf6e32adb152fada6d08e54447f7d2e9370ea883/declaration.txt

The "-O" part is important. It's a flag: an extra option that indicates to the program we want to do something different from its default operation. By default, curl will show the file. By using the -O option, we can instruct curl to download the file. In this case, it will be saved as declaration.txt.

Open the project folder in your favorite editor. Make sure to open the whole folder (and not just the declaration.txt file) by clicking the folder and choosing "Open". You will see the list of files in the sidebar.

Let's make a Python script called count_words.py. We can create our Python script in this same folder by clicking the "new file" icon in Visual Studio Code, or we can do it from the command line:

touch count_words.py

The touch command is meant to change the date and time the file was last accessed. A side effect of the command is that it will create a new file if it doesn't already exist, which is what we use it for.

We're going to write some very basic Python code to open the file, read it in, then show us the first 10 words of the file. Here's the code:

text = open('declaration.txt').read()
words = text.split()
print(words[:10])

This code demonstrates why I like Python: it's terse, and contains just the information we need without being "magic".

The first line opens the file and reads the contents. It stores this in a variable we define called "text". Note that we can call it anything we like, as long as we use the same name to refer to it later.

We didn't specify what type the variable has. In Python each variable will have a type when the script is running. Here it's going to be a string: a container that holds text. It also supports Unicode, so we can store complex strings and Emoji's.

The second line takes the text we read in and splits it. Since we didn't tell it on what to split, the string will be split on whitespace. The result will be stored in the words variable (again, we choose that name ourselves).

The type of the words variable is a list. It's the most useful type in Python: lists are the core of any data processing job. Complex jobs will require us to iterate over lists of directories, which might contain lists of files, where each file is a list of lines each containing lists of words, which consist of a list of characters... Even images can be regarded as a list of numbers defining the red, green and blue values making up the pixels of the image.

We can do a lot with lists. The last line in our script just prints out the first 10 words of the list, using the [:10] syntax. This means print from the beginning of the list, until (but not including) the 10th element.

Save the script, then run it by cd-ing to the correct directory if you're not there already and typing:

python3 count_words.py

You should see the following output:

['When', 'in', 'the', 'Course', 'of', 'human', 'events,', 'it', 'becomes', 'necessary']

If not, check every letter of the script carefully. If you get a FileNotFoundError it means that Python can't find the file. Make sure it is in the same directory and has the same name as written in the script.

To actually count the words we need another data structure. We need a way to map each word to a value: the number of times we encounter that word in the text. In other words, we want to have something that looks like this:

of     78
the    76
to     64
and    55
our    25
their  20
has    20
for    19
in     18
He     18

In Python we can use a dictionary to contain this mapping. It stores a key and an associated value. In our case, the key is going to be the word, and the value is going to be our count.

Here's the full code:

text = open('declaration.txt').read()
words = text.split()
word_count = {}
for word in words:
    count = word_count.get(word, 0)
    count += 1
    word_count[word] = count
print(word_count)

The initial two lines are the same as before. The third line creates a new variable called word_count that is an empty dictionary.

Then we encounter our main loop. This block iterates over each word and increases the count for it. In Python, a block is indented (has extra spaces). It's recommended to use 4 spaces. All indented lines are executed in the loop. Note that the print statement at the bottom is not indented. We don't want it to happen after every word, but instead after all words have been counted.

Let's look at the lines in our loop:

count = word_count.get(word, 0)

The first line looks up the word in the word_count dictionary. To get a value from a dictionary in Python we can also use square brackets (e.g. word_count[word]). However, this will fail if the word is not available. That's why we use get to provide a default value of 0 if the word can not be found. So after this line, the count is either the number of times we encountered the word, or 0 if we haven't seen it yet.

count += 1

This line increases the count by 1. It's shorthand for count = count + 1. It takes the count, adds one to it, then stores it back into the count variable.

word_count[word] = count

Finally we'll store the updated count back into the word_count dictionary, under the key contained in the word.

After the loop is done we print out the list of words. However, you'll see that this format is not really useful:

{'When': 1, 'in': 18, 'the': 76, 'Course': 1, 'of': 78, 'human': 1, 'events,': 1, 'it': 5, 'becomes': 2, 'necessary': 2, 'for': 19, 'one': 1, 'people': 2, 'to': 64, 'dissolve': 1, 'political': 2, 'bands': 1, 'which': 10, 'have': 11, 'connected': 1, 'them': 11, 'with': 9, 'another,': 1, 'and': 55, 'assume,': 1, 'among': 5, 'Powers': 1, 'earth,': 1, 'separate': 1, 'equal': 1, 'station': 1, 'Laws': 6, 'Nature': 1, "Nature's": 1, 'God': 1, 'entitle': 1, 'them,': 3, 'a': 15, 'decent': 1,...

The words are ordered not by the count, but by the order in which we encounter them. We'd like to see them in a more useful format, so let's sort the dictionary.

Actually, we can't really sort a dictionary. Instead, we can get a representation of the dictionary with the keys, sorted.

Remove the last print statement and replace it with the following code:

word_count_list = sorted(word_count, key=word_count.get, reverse=True)
for word in word_count_list[:20]:
    print(word, word_count[word])

The first line here is a bit tricky. sorted is a built-in Python operation that sorts the input and returns a list. By default, if we give it a dictionary it will sort the keys alphabetically. Instead, we provide a key argument to indicate on what to sort. Here, that is the value of each item, which we can write by asking it to get the value for each item.

This would start with the smallest values, so we use reverse=True to tell Python to reverse the ordering.

Once we have this list, we print the first 20 words of it. We loop over them, then call print with both the word, and the word count, retrieved by calling word_count[word].

This is what the output looks like:

of 78
the 76
to 64
and 55
our 25
their 20
has 20
for 19
in 18
He 18
a 15
these 13
by 13
have 11
them 11
which 10
that 10
all 10
is 10
with 9

Obviously English stop words will be the most frequent words appearing in the top of the list. We can filter those out if we want (if we have a list of stop words), or we can just print more of the list (e.g. 50 items) to see more interesting words.

Another issue is that commas and periods will be regarded as part of the word, messing up the count. We want to remove all special characters and only worry about the words. In addition we should consider the casing: Python considers the word "People" and "people" as two distinct words. We'll turn everything into lower case so words are countly correctly.

At the top of our file, add the following code:

import string

translator = str.maketrans('','', string.punctuation)

This imports an external module called "string" that contains a list of punctuation. The next line creates a translator which is a Python construct allowing us to remove all punctuation.

To use this, add the following code right before the count = word.get line:

word = word.translate(translator).lower()

Give it the same indentation (4 spaces) as the lines below it. This takes the word and removes all special characters.

Run the script again. We'll see that the word "people" is now the most frequent word after all common stop words.

To save this information in a useful format we'll use the csv.writer to write out the data as comma-separated values: basically a simple text-based format that we can open in Excel, Google Docs, NodeBox, Processing or D3.

We'll add an import at the top of the file:

import csv

Then, add the very end of the file, add the following code:

output_file = open('words.csv', 'w')
writer = csv.writer(output_file)
writer.writerow(['word', 'count'])
for word in word_count_list:
    writer.writerow([word, word_count[word]])

This opens a new file, called words.csv. Note the extra argument with the 'w'. This is necessary to tell Python to open the file for writing. This makes sure we have the correct permissions to write to a file. The next line takes this file (often called the file handle) and turns it into a CSV writer. This is a special object that has a method called writerow allowing us to write CSV data.

CSV files look like this:

word,count
of,78
the,77
to,65
and,56
for,28
our,26
their,20
has,20
in,19

The CSV writer can output data in this format, making sure that the system doesn't mess up if our actual word contains a comma.

Run the script again. You should now see a words.csv file appear. Open it up in your editor and check that it contains the list of words and counts.

Congratulations! You created your first Python data processing job. Here's the finished Python code, for reference.

Google Cloud OnBoard Eindhoven

2017-09-22T00:00:00Z

On Wednesday September 20th I participated in the Google Cloud OnBoard event at the High Tech Campus in Eindhoven.

The event was an introduction in the features of the Google Cloud infrastructure: everything from hosting your own apps, storage and databases to working with big data and machine learning.

The event was organized by Google and presented by Matthew Feigal from Google and Mark Crump from QA, a Google training partner.

Introduction

The Google Cloud platform is big, but can be broken down in four sections:

Compute: managing the machines that run your applications. Ranging from Compute Engine, which is a VPS, to App Engine, which is a Platform as a Service (PaaS).
Storage: managing how you store and access data. This covers file storage (Cloud Storage), Cloud SQL and Bigtable for big projects.
Big Data: asking questions about large datasets. Use BigQuery to perform SQL-like queries on terabyte-size datasets, Pub/Sub for messaging, Datalab for an IPython Notebook in the cloud.
Machine Learning: access to Google's infrastructure for your own machine learning jobs using TensorFlow, or pre-built machine learning apps like Cloud Vision, the Natural Language API or the Speech API.

Monitoring all of that data happens through StackDriver, an application to monitor, log and debug applications.

Infrastructure

Google's physical network is impressive. They essentially built their own internet, including undersea cables and dark fiber, so that once you get on the Google network everything is super-fast.
The primary Europe data center (europe-west1) is located in Belgium, at St. Ghislain.
You can ping the Google data centers to get an idea for latency.
For large customers, they offer to peer so you can connect directly to their network. In Amsterdam, you can use Equinix to peer.
360° video tour of the Google Data Center. Spot the security guard in every shot.
There is a book about how Google runs production systems.

Projects and Access

Everything in Google Cloud Platform is managed by project. A project contains all infrastructure you want to set up: server, storage, logging, ... . It also unifies billing. A project has a visible name and an internal ID that needs to be globally unique.
You can use IAM (Identity and Access Management) to control who has access to what. This can be really granular.

Privacy / GDPR

Google complies with available ISO and HIPAA standards.
Google intends to fully comply with the European General Data Protection Regulation (GDPR), once they become available.
You can choose in which region your data is stored (North America, South America, Europe, Asia, Australia).
All data is encrypted at rest. If you don't trust Google, you can also provide your own custom encryption keys.

Interacting with Google Cloud Platform

You can access GCP (Google Cloud Platform) using different methods:

Cloud Console: a web interface to Google Cloud.
Cloud SDK: terminal-based tools for accessing services and deploying apps.
Cloud Shell: from the web interface, you get free access to a Linux machine that contains all tools pre-installed. Super-useful.
REST-based APIs: so you can create services on-demand. Use API Explorer to get an overview and try out all accessible APIs. Libraries are available for Java, Python, JavaScript, PHP, .NET, Go, Node.js, Ruby, Objective-C, Dart.
Mobile App: for managing services from iOS or Android.

Compute

App Engine

Easiest way to build an application that doesn't require maintenance and works online.
Similar to Heroku.
Two types of environments:
- Standard: autoscaling, free daily quota, limited language support (Java, Python, Go). Has some restrictions to enable scaling: no writing to local file system, for example. Automatic shutdown if the app is not used, saving cost.
- Flex: works with containers, which you can inspect and log in to. Supports every possible language.
Provides extra services like cron jobs, task queues for rendering or batch processing, search, logs.
Example: Crowdsite, dutch marketplace for design & creatives.

Compute Engine

VPS, like Linode or Digital Ocean.
Different variants: high CPU, high memory, even GPU machines.
Running a single 512MB server is free!
Fast to spin up, per-minute billing.
Use load balancer to distribute load between multiple machines.
Supports live migration, so the server keeps running as it is migrated to a different host.
You can set and access metadata for bootstrapping a server. The server has access to http://metadata.google.internal/.

Container Engine

Used for virtualization at the operating system layer. Separates operating system from application code and dependencies.

Containers are used everywhere in Google. Any application runs in a container: Search, Mail, Maps, ... It's the best way to keep dependencies of different applications from messing with each other.

Consistent environment for development, testing, and production. Simple to migrate and move around.

Docker and rkt are the most popular container implementations.

Kubernetes

If you have multiple containers, you might want to orchestrate (manage) them. You can use Kubernetes for that. It automates deployment and scaling.

Kubernetes specifies a configuration file that contains the requirements of your app: project folder, RAM, disk, CPU and how many replicas you want to have running. The system then makes sure you have that many containers always available. In Google this system is called Borg.

Containers lie between directly using a VPS, and the Platform-as-a-Service. You can run everything you want, but Google manages scaling and failover.

Example: Philips Hue uses containers for scaling the infrastructure controlling the lights in your home. They use Apigee Edge for managing access to their API, and Redis for storage.

Container Engine is the hosted version of Kubernetes, running on Google Cloud Platform.

Cloud Functions

Run a single function in the cloud, using JavaScript.
No servers to manage, billing in 100ms intervals.
Functions that respond to events (e.g. subscribe button, IoT event).
Works with Firebase.
Perfect for microservices.

Cloud Networking

Once you have many systems, you can connect them in a virtual network (VPC). You can set this up using custom routing, DNS, load balancing.
All nodes in the network (e.g. 192.168.20.0/24) can be in different regions of the world.
Google also provides Cloud CDN, a content delivery network.
Google Domains is a domain name registrar for buying and managing domain names. Not available in Belgium / The Netherlands yet.

Cloud Endpoints

Proxy for your APIs.
Supports access control, caching, monitoring.

Cloud Deployment Manager

Create templates to deploy your resources in a repeatable manner.
Like Puppet or Chef.

Storage

Cloud Storage

Like Amazon S3, but cheaper. Similar structure: files are stored in Buckets.
Access the data from the command line using gsutil or through the web interface. You can also transfer data from S3 directly into Cloud Storage.
You can choose where the data is located: Europe, Asia, America, or multi-regional.
Supports "nearline" and "coldline", for archival data that is infrequently accessed. Much cheaper, but more expensive to access. Unlike Amazon Glacier, access to the nearline and coldline data is immediate.
Often used as a first step to store data that can be processed by machine learning jobs, for example tables for BigQuery or images for Cloud Vision.
Can also serve HTTP/HTTPS traffic. If you have a static site, this might be all you need.
Good for storing images, media files, backups.

Cloud Datastore

NoSQL database supporting billions of rows.
Schemaless access.
View and edit the data in the web console interface.
Useful for simple applications but you often need some sort of schema for more complex operations.
Good for product catalogs, user profiles.
Tutorial application with App Engine and Cloud Datastore

Cloud Bigtable

Wide-column NoSQL store for terabyte to petabyte-scale applications.
Used in Gmail, Google Analytics.
Similar to Cassandra. Accessed using the HBase API (Hadoop).
Good for "flat data": time series, AdTech, financial data, IoT sensor data, computer logs.

Cloud SQL

Managed MySQL and PostgreSQL database as a service.
Automatic replication, backups, scaling.
Easy to integrate with App Engine or Compute Engine instances.
Good for typical SQL workloads: relational data, customer orders.

Cloud Spanner

Horizontally-scalable, strongly consistent relational database.
Based on the Spanner paper.
Global consistency: very hard to do! Uses atomic clocks in all servers.
Good for large-scale database applications (> 2 TB).
Used in GMail for storing metadata.

Cloud Source Repositories

Managed private Git hosting, like GitHub enterprise.
Can mirror other Git repositories, like from GitHub for example.
Supports webhooks so that actions happen when a commit is pushed.

StackDriver

Used for monitoring, logging, debugging, error reporting on your infrastructure.
Nice visualizations.

Big Data and Machine Learning

Running big data operations on Google Cloud means you just need to focus on the application. You don't have to manage servers or acquire costly GPUs.

Cloud BigQuery

An analytics database that provides near real-time interactive analysis of massive datasets (hundreds of TBs).

You throw your data at it (CSV, JSON, realtime streams), write a SQL query, and get a result back.
Based on the Dremel paper.
During the presentation, we saw queries on the full Wikipedia data set (100 billion rows) in a few seconds.
Can query custom datasets, or public datasets such as GitHub, Open Images, World Bank.
Good for medical data, analytics, ...

Cloud Pub/Sub

Managed messaging queue, like RabbitMQ or Kafka.
Includes support for offline consumers.
Good for Internet of Things (IoT), marketing analytics, connecting different parts of your application.

Cloud Dataflow

Create data pipelines for ETL and run them on a scalable, managed infrastructure.
A managed installation of Apache Beam
Integrates with all other cloud services. BigQuery, Cloud Storage, Pub/Sub.
Write dataflows in Java and Python.
Example use case: find the top watched videos at the moment, in 5 minute windows.

Cloud Dataproc

Managed Hadoop
Fast to create your own clusters (90 seconds).
Good for analyzing large data sets (e.g. logging data) stored in Cloud Storage.

Cloud Datalab

Jupyter (IPython notebook) in the cloud.
Run and manage data tasks and machine learning in a cloud environment, with fast access to Cloud Storage, machine learning APIs, etc.
Free (but you pay the costs of accessing other cloud services).

Cloud Dataprep

Service for exploring, visualizing and cleaning up your data.
Similar to Google Refine / OpenRefine, but running in the cloud.

Machine Learning

Google open-source TensorFlow, a widely-used tool for machine learning. They now provide tools to run TensorFlow jobs in the cloud, as well as pre-trained models for computer vision, language analysis and translation, and text-to-speech.

Rokesh Jankie, a Google engineer, gave the best one-line introduction to AI I've ever heard: "You create a function, only we don't know yet the formula of the function." Instead of linear functions, we use neural networks, which can handle complex transformations and noise better. He demonstrated this using the TensorFlow Playground. He also showed the Cloud Vision Explorer, showing an interactive demonstration of the capabilities of the Cloud Vision API. He also showed WaveNet, a generative model for the generation of text-to-speech. He demonstrated how computer vision can be used to find disease patterns in images of eyes.

A few use cases:

Classification / Regression: forecasting, customer churn analysis ("why are my customers leaving?")
Recommendations: content personalization
Anomaly detection: detect fraud, log anomalies in metrics or sensors
Image analytics: identify damaged shipments, detect explicit (adult/violent) images.
Text analytics: call center log analysis, language identification, sentiment extraction.

Cloud Machine Learning Engine

For custom machine learning jobs.
Google provides infrastructure for training your model.
Once trained, your model can run on user's computer or smartphones.

Cloud Vision API

Analyze images with a REST api.
Supports face detection, label detection, sentiment analysis, text extraction (OCR).
Returns bounding boxes of faces (and facial features), objects in JSON.
COCO is an image dataset that has labelled and segmented images for use in image recognition. It is available under a Creative Commons license.

Cloud Speech API

Recognizes over 80 languages.
Can return text in real-time (streaming).

Cloud Natural Language API

Extract information about people, places, events mentioned in text: news articles, papers, blog posts.
Sentiment extraction.
Can be used for Chatbots.
Ocado, an online grocery store in the UK, used it to triage customer requests into urgent delivery-related messages and other messages.
Dealing with sarcasm is still difficult. You can tune the parameters of the network.

Cloud Translation API

Translate strings between different languages.
Detect language of input text.

AutoML

Selecting machine learning architectures is a difficult and time-consuming task.
AutoML is an area of research that tries to automate this process, by having machine learning algorithms figure out the best machine learning algorithms.
One way to do this is to have neural networks compete with each other.

Next Steps

Learning Resources

Official Google Cloud Platform documentation site.
GitHub page with tutorials.
Qwiklabs: self-paced training labs.
Google also offers training and certification.

Community

In the Netherlands there is the Google Developers Group. There is also a Meetup Group.
In Belgium there is the Brussels GDG.

Making of my PhD Thesis Generator

2016-06-29T00:00:00Z

On June 15, I received my PhD in the Arts from the University of Antwerp / ARIA. My research was about the impact of generative design tools. I researched how generative design tools (specifically NodeBox) affect the quality and creativity of designers, and whether they were more efficient and self-reliant.

A PhD in the arts lies somewhere between scientific research and artistic practice. It is at times both, and at other times neither. One day you'll be performing A/B tests on users, the other days you're knee-deep in artistic projects. I wanted to represent this battle between two sides in the thesis as well. In line with the message of the PhD I didn't simply want to create a thesis, but a thesis generator: a system that could generate endless variations of the thesis (keeping the content intact but dynamically generating the design).

I've opted to created "the slider": an interactive element that allows you to choose for yourself whether you want an artistic or scientific thesis. The design is adapted to match, and every design will be unique.

Go generate your own thesis

Some designs evolve over multiple pages, gradually changing background elements from page to page. Flipping through the thesis results in interesting animations:

I had a selection of these theses printed. Each design used different font sizes and margins, meaning layouts had to be recalculated, taking typesetting issues like widows and orphans into account. The total number of pages varied between 140 and 180, meaning that the thickness of the book varied as well. I added routines to calculate the width of the spine and generate different-sized book covers based on those calculations.

The text of the thesis was written in Workflowy. It was imported automatically and converted to Markdown. The thesis generator was built in Go. It doesn't use any additional packages apart from the Go standard library. This means that every single byte of the PDF is generated by my own software. Here are some of the aspects that were implemented, from high-level to low-level:

Web Server
Style interpolation
Workflowy import
Automatic figure generation (from inline NB1-style Python code)
Markdown parsing
Text layout (justification)
Layout calculation
Dynamic background generation
Automatic TOC and bibliography generation
Separate cover / body generation
Font parsing (to determine widths, taking kerning into account)
Color handling (HSL / RGB calculations)
PDF object generation (pages, outlines, font and image embedding)
PDF primitives (dictionaries, arrays, xrefs, ...)

The low-level PDF handling and layout and typesetting calculations might be interesting as a library to release in the future. In addition, higher-level concepts like style interpolation might be interesting as well; currently they are connected to the thesis.

Go generate your own thesis

Random Compositions using Overtone

2013-10-11T00:00:00Z

While learning Overtone I've created a composition that randomly downloads sounds from freesounds.org and makes beat-like soundscapes with them. (First composition starts at 1"20.)

If you want to try this yourself, start by downloading Leiningen from leiningen.org.

Deconstructing Nature - Day 5

2012-09-08T00:00:00Z

My colleague Werner and I are holding a challenge this week to produce one visual work each day. I'm working around the theme of "Deconstructing Nature".

Day 1 — Day 2 — Day 3 — Day 4 — Day 5

For the last day I wanted to do something with human nature. And there is no better place on the internet to watch human nature decompose than 4chan (Warning: Not Safe For Work!). The website is a huge image board on a large number of topics ranging from Anime to Politics.

And then there's the "random" image board, also known as /b/, also known as the cesspool of the internet. I wanted to visualize the streams of data coming out of there, and with 4chan recently providing an API to their data, this might actually be feasible.

I use a veritable potpourri of Python libraries to bring everything together:

In the end, it was very hard to do something with images that are often gross, sometimes shocking and mostly bizarre. So I decided to abstract the representation:

Again, this is a visualization of the random image board. It would be interesting to see visualizations of the other thematic boars to see if they show different patterns.

Other sketches:

The source is available in the Deconstructing Nature GitHub repository.

Deconstructing Nature - Day 3

2012-09-06T00:00:00Z

My colleague Werner and I are holding a challenge this week to produce one visual work each day. I'm working around the theme of "Deconstructing Nature".

Day 1 — Day 2 — Day 3 — Day 4 — Day 5

Inspired by Werner's work I wanted to do something with swarms. For another project we're talking about creating a living wallpaper which I also wanted to integrate here. And I wanted to make something that was animated and a little bit interactive.

I decided to create an animated swarm in a browser, using trails to create a wallpaper. I used the excellent three.js library to access WebGL.

Here's the result from day 3. Click the image to view it live (Chrome / Firefox).

Keys:

Hold Spacebar: Direct the swarm to the center.
C: Clear the screen.

The source is available in the Deconstructing Nature GitHub repository.

Deconstructing Nature - Day 2

2012-09-05T00:00:00Z

My colleague Werner and I are holding a challenge this week to produce one visual work each day. I'm working around the theme of "Deconstructing Nature".

Day 1 — Day 2 — Day 3 — Day 4 — Day 5

For the second day I wanted to do something physical. I was thinking about the act of deconstructing – breaking something down to its simplest components. "Simpling".

This brought me to the idea of microscopy. I discovered that you can do extreme macro-photography by installing the lens backwards on a camera.

I did some experiments with my aging DSLR, but results were not as sharp as I hoped. However, I discovered that I can "attach" a regular lens to my smartphone, producing very detailed images. I used this technique to create the final image.

Here's the result from day 2:

Here are some other images from this session:

Deconstructing Nature - Day 1

2012-09-04T00:00:00Z

My colleague Werner and I are holding a challenge this week to produce one visual work each day. I've given myself the theme of "Deconstructing Nature", which you can interpret as wide as you want.

Day 1 — Day 2 — Day 3 — Day 4 — Day 5

For the first day I wanted to design the layout of a space – specifically a garden – according to patterns occurring in nature. I thought about:

The spread of forest fires
The intricate nerve pattern of plants
Patterns of underground movement: worms, ants, tree roots, ...

I liked the idea of constructing something above ground that followed movement patterns below ground. I stumbled a lot trying to model the movement of worms, but in the end used a tweaked boids algorithm to produce strange natural/artifical hybrids.

Here's the result from day 1:

Here are some details of the work:

The work is made in NodeBox 1 using a tweaked Boids algorithm (created by Tom De Smedt).

Here are some other sketches I made:

The source is available in the Deconstructing Nature GitHub repository.

And finally, you can watch a timelapse of the construction.

Starting web projects using Flask, virtualenv, pip

2012-08-16T00:00:00Z

Flask is an awesome framework for building lightweight Python web-apps using a minimal amount of boilerplate. This tutorial discusses how to start a new Flask project and how to deploy it on a server.

These instructions are tested on Mac OS X 10.8. All code is available under the GitHub helloflask project.

Bootstrapping your project

One server can host multiple running web applications, and they probably all use different versions of the underlying frameworks. To manage these dependencies, we'll create a virtual environment where we install the dependencies locally, together with the project instead of globally. In Python, we'll do this using virtualenv and pip. These tools provide a clean way to manage the dependencies of different Python projects.

Install virtualenv

Virtualenv creates the virtual environment. Initializing a virtual environment automatically installs pip as well, which does the Python package installs. We'll install virtualenv locally, and again on the server. But first, let's install it on your machine:

sudo easy_install virtualenv

Setup a new project using virtualenv.

To start our new project, we're going to use virtualenv to setup the directory for us:

virtualenv helloflask

This creates a folder "helloflask" that includes an isolated Python install. This project will not look at global site-packages, and packages installed there don't affect the global Python install. Hence the "isolated".

Start working on your project

Each time you work on this project, you need to activate the virtual environment. From a new Terminal window:

cd helloflask source bin/activate

You can check that your environment is active because virtualenv puts the name of the app before the

Any packages you install now using pip or easy_install get installed into helloflask/lib/python2.7/site-packages.

Install Flask

We've activated our virtual environment. This environment is devoid of any installed packages, so let's install Flask:

# Make sure you've done "source bin/activate" first pip install flask

This installs Flask, Werkzeug (a utility library) and Jinja (a templating framework).

Write some code

Create a file helloflask.py that contains the following application:

from flask import Flask app = Flask(name)

@app.route("/") def hello(): return "Hello World!"

if name == "main": app.run()

Run the application:

python helloflask.py

You should be able to surf to http://127.0.0.1:5000 and see a cheery "Hello World!" message.

To stop the server process, press control-C.

Learn Flask

Browse through the excellent Flask documentation for more information on how to build the actual application. This is not covered here.

List your requirements

When deploying to a webserver it is important to register which requirements we need. To do this we freeze the installed packages and store this setup in a requirements.txt file:

pip freeze > requirements.txt

This writes a plain text file that contains the names of the required Python packages and their versions, for example Flask==0.9. We'll use this file later when we're setting up our server.

The WSGI script

When we're running our application on a browser, we'll need a "run script" that tells the server how to setup your application. In other words, the webserver doesn't run your application directly like we did from the terminal, but uses WSGI to load the application into a separate process. In this WSGI script we can tell the server to use virtualenv to load our setup.

Create a new file called application.wsgi with the following contents:

import os, sys

PROJECT_DIR = '/www/helloflask.enigmeta.com/helloflask'

activate_this = os.path.join(PROJECT_DIR, 'bin', 'activate_this.py') execfile(activate_this, dict(file=activate_this)) sys.path.append(PROJECT_DIR)

from helloflask import app as application

Note that this file refers to a (non-existent) /www/helloflask.enigmeta.com directory. We'll create this directory on the server in a minute.

Version your files

To make development and deployment easier we should really use version control. Here, we'll use Git to manage our files. You should already have a working Git install. At the very least, make sure you've configured your full name and email address:

git config --global user.name "Your Name" git config --global user.email "your.email@example.org"

Initialize Git in our project directory:

git init

Because virtualenv installs a lot of unneeded files, we should ignore them. We need .gitignore to do this for us. Because .gitignore is a hidden file, we could use an editor like nano to change it:

nano .gitignore

An initial .gitignore file should look like this:

# Virtualenv .Python bin lib include *.py[cod]

# Mac OS X custom attribute files .DS_Store

Now add everything to your initial commit:

git add . git status

Git status should list all files. This should be helloflask.py, application.wsgi, .gitignore and requirements.txt. With those three files added to the Git index, let's commit them:

git commit -m "Initial commit."

Store the project in a central repository

In this example we'll use GitHub to store our project. This gives us a central place where we can push files to (our machine → GitHub) and pull files from (GitHub → webserver). You can use BitBucket which provides unlimited private repositories, or a custom setup using the webserver itself.

Create a GitHub account if you don't have one yet. Then create a new project. Call it "helloflask". Then, to push to this application, setup GitHub as the origin:

git remote add origin https://github.com/USERNAME/helloflask.git git push -u origin master

This is all we need to do on the client side. Let's look at the server for now.

Setup the server

In this example we're using Apache 2 as the web server.

We're going to assume we're running our app on its own domain or subdomain. This means we'll use a virtual host with its own configuration.

This is on a virtual private server (VPS), such as from Linode (referral link).

Let's set up the directory structure first. It will look like this:

/www # Root of all sites on this machine helloflask.example.org # Your full domain and subdomain helloflask # Python project folder logs # Server access and error logs

To set up this directory structure:

# Install virtualenv sudo su # You probably need to be root to do this. apt-get install python-virtualenv

# Create the directory structure mkdir -p /www/helloflask.example.org cd /www/helloflask.example.org mkdir logs

# Clone the project git clone https://github.com/USERNAME/helloflask.git

# Initialize virtualenv and install dependencies virtualenv helloflask cd helloflask pip install -r requirements.txt

Here's a complete template Apache site configuration that can go in /etc/apache2/sites-available/helloflask.example.org (change the last part to reflect your domain):

<VirtualHost *:80> ServerName helloflask.enigmeta.com

WSGIDaemonProcess helloflask user=www-data group=www-data threads=5 WSGIScriptAlias / /www/helloflask.enigmeta.com/helloflask/application.wsgi

<Directory /www/helloflask.enigmeta.com/helloflask> WSGIProcessGroup helloflask WSGIApplicationGroup %{GLOBAL} Order deny,allow Allow from all LogLevel warn ErrorLog /www/helloflask.enigmeta.com/log/error.log CustomLog /www/helloflask.enigmeta.com/log/access.log combined

Activate the virtual host:

a2ensite helloflask.example.org service apache2 reload

You should now have the site running under helloflask.example.org (replace this with your domain).

Automate deployments using Fabric

Manually re-uploading the code and restarting the server gets tedious after a while. Instead, we can automate this by using Fabric.

Fabric is a command-line utility that runs a fabfile containing instructions on how to deploy to a server. It can automate any server task, not just deployment.

Install Fabric using:

pip install Fabric

Here's an example Fabric script (change the relevant variables at the top of the script):

from fabric.api import env, run, cd

USERNAME = 'root' SERVER = 'helloflask.example.org' APP_NAME = 'helloflask' PROJECT_DIR = '/www/%s/%s' % (SERVER, APP_NAME) WSGI_SCRIPT = 'application.wsgi'

env.hosts = ["%s@%s" % (USERNAME, SERVER)]

def deploy(): with cd(PROJECT_DIR): run('git pull') run('source bin/activate; pip install -r requirements.txt') run('touch %s' % WSGI_SCRIPT)

Let's go over the commands we're running:

git pull # Get the new source code from GitHub bin source/activate # Activate the virtual environment (needed for the next step) pip install -r requirements.txt # Check if there are new required packages and install them. touch helloflask.py # Let Apache reload your application.

Save this in a file called fabfile.py in the root of your project and add it to version control:

git add fabfile.py git commit -m "Add deployment automation script."

Subsequent deployments can be done using:

fab deploy

Fabric shows the output of the server so you can check when things go wrong.

Update workflow

Now that you have a fully working install, you can keep updating the application with a minimum of effort:

Run the server locally.
Change helloflask.py locally and go to http://localhost:5000 to see the results.
If you're happy with the results you can commit the changes and push them to GitHub.
Run fab deploy to update the application on the server.

Teaching lists to designers

2012-05-04T00:00:00Z

I've been teaching generative design using NodeBox 1 since 2004. NodeBox 1 is a Python IDE for making generative graphics. With my team at EMRG I've been teaching one- or two-week workshops for graphic designers, resulting in a poster or animation at the end. You can see some of the process / results at the Workshop Blog.

The most important insight I've discovered during the workshops is to teach people data structures as soon as possible. Often I teach this before learning about conditions or loops. Data structures are absolutely essential to let students progress from the "beginner" stage to the "competent" stage.

Before, I learned the "building blocks of programming": variables, loops, conditions, functions. But without a way to store and manipulate data, none of these constructs are useful.

One week is too short to learn about all data structures so we only teach lists. Lists can be used for everything:

the keywords of a web page
the points of a path
the installed fonts
the images in a folder

All of these require lists. Once we have a list what can we do with it? We can:

Iterate a list

for pt in path.points:
    ellipse(pt.x + random(-10, 10), pt.y + random(-10, 10), 2, 2)

Sort a list

import glob
import Image, ImageStat

def image_brightness(fname):
   img = Image.open(fname).convert('L')
   stat = ImageStat.Stat(img)
   return stat.mean[0]

icon_files = glob.glob('icons/*.png')
sorted_icons = sorted(icon_files, key=image_brightness)

Choose elements from the list

from random import choice
    all_words = open('/usr/share/dict/web2').readlines()
    old_word = choice(all_words).strip()
    new_word = choice(all_words).strip()
    print '%s is the new %s.' % (old_word.capitalize(), new_word)

For some students we also teach dictionaries (maps) if they need it to do lookups. But lists are absolutely essential.

Before teaching data structures students were struggling in going from their idea to a solution. By teaching lists we've seen a huge increase in self-reliance. They can now express their ideas in terms of a programming construct. Have a set of Wikipedia articles you want to visualize? Use a list to store them and iterate through them. Have an idea for a font where the points are randomly wiggled? Get the list of points from a glyph shape and add a random value.

I think most designers never learn to program because all the teaching material is about variables, conditions and loops. As programmers we know that programming is fundamentally about manipulating data. But designers don't know this! By teaching data structures as soon as possible they understand why knowing coding can be useful for them as well.

The Alternate Universe of the Mac App Store

2012-03-18T00:00:00Z

When trying to reinstall my MacBook Air, I tried going through the Mac App Store to download my applications. It seems like the store operates in an alternate universe:

Profilitics

2012-02-24T00:00:00Z

The 2012 American elections promise to be exciting. Candidates throwing mud at eachother, and attacks becoming increasingly more personal. But how different are the candidates really?

As part of the Generator.x 3.0 workshop by Marius Watz, I worked together with Lieven Menschaert to produce three-dimensional lasercut heads detailing what really goes through the minds of the republican and democratic candidates.

Using generative software, we map the democratic and republican candidate on a variety of layers:

Keywords layers: containing keywords from quotes of political candidates on important issues.
Campaign contributions: amount of contributions from various sectors.
Regional popularity: a map of the states linked to a certain party.

The project uses an alpha version of NodeBox 3, developed simultaneously with the project.

City in a Bottle

2011-11-01T00:00:00Z

"City in a Bottle" is an interactive, generative forest with computer-generated trees, plants and animals.

Together with artists, musicians and programmers, we created a project bordering between art, science and gaming, funded by the Flemish Audiovisual Fund.

http://www.cityinabottle.org/

Word Maze

2011-06-01T00:00:00Z

Made for the "Travelling Letters exhibition" in 2011 Vilnius, Lithuania, I developed a set of words around the concept of "transit", disguised as generative mazes.

NodeBox 2 is a Graph

2011-05-01T00:00:00Z

In this post, I'll explain how NodeBox 2 is build as a big graph evaluator and describe how the program arrives at its visual output. I will detail the difficulties in marrying the functional aspects of NodeBox 2 with Processing's imperative nature.

NodeBox 2 as a functional language

NodeBox 2 is a program where operations are presented as a graph of nodes. Each node fetches data from its inputs, calculates a new result and sends it off to its outputs. In the end, the output of one node gets displayed (this node is marked in the screenshot with a yellow rim).

The underlying data structure is a directed acyclic graph. This is a graph where all nodes have a direction, and where cycles between nodes verboten.

Each node contains a "cook" function. The output of the node (which is stored in its output port) is the return value of this function. We mark one of the nodes as the rendered node. This node is the one we want to cook.

From the rendered node, find its dependencies. These are the nodes connected to input ports of this node.
For each dependency:
- Find its dependencies and update them, recursively.
- Cook the node. This executes the cook function and puts the return value in the output port, ready for the dependent nodes to pick it up.
Cook the rendered node.
Draw the result of the rendered node.

This approach has a number of benefits:

There is one clear endpoint, the rendered node. By changing the rendered node, users can look at the output of a node anywhere in the graph.
The graph is determinate. Cooking the network always gives the same result.
Unchanged parts of the graph can be cached. We can determine which nodes we need to recalculate whenever something changes by looking at its dependencies. We try to be as lazy as possible. Doing the least amount of work needed is good for speed.

In other words, NodeBox 2 takes some aspects from functional programming, such as "pure" functions and function composition, and makes them visual. Now that I think about it, this explains my fascination for Clojure.

Processing is a state machine

We'd like to integrate Processing into the network. Processing provides an API for drawing, interactivity and animation.

The design of Processing is quite different. Processing is designed as a state machine. This has a number of effects:

Here is a simple Processing example that demonstrates its stateful nature.

  void draw() {
    if (mousePressed) {
      fill(255);
    } else {
      fill(0);
    }
    ellipse(mouseX, mouseY, 50, 50);
  }

And here's what it looks like:

Note that we don't specify the fill color in the ellipse. Instead, we set the fill color. All commands that draw something, from that point on, will use this new fill color. Also note that the ellipse command immediately draws something on the screen.

In other words, all Processing commands have side effects. We don't care about their return value, but we call them because they change something in the world. Processing is an imperative language.

Compare this to the typical code of a node:

  def cook(self):
    p = Path()
    p.rect(self.x, self.y, self.width, self.height)
    p.fill = self.fill
    return p

This code doesn't draw anything! All it does is create a Path object with the right geometry and color and return it. It is free of side effects. You could call this code many times, or not at all if you already have the cached result.

The basic problem is as follows:

We can no longer be sure that a node has no side effects. Quite the contrary! Processing is all about managing state.
There is no clear endpoint or "rendered node". Instead, the render is a result of calling the commands.
The order in which nodes are cooked is important.
Caching doesn't work since the functions have to be called for them to have an effect.
There is no data involved! In the Processing example, nothing gets passed on.

Processing → Graph

Let's try converting our simple Processing example into a node network.

We'll start simple by drawing a rectangle.

This node has a number of connection points, or ports that can be connected. As in the example, we'll connect the X and Y coordinates to the mouse.

Now, the rendered node comes into play. Since we have more than one node, we need to know which one to render. Here, we want to see the result of drawing the rectangle. The mouse node is a dependency.

We haven't set a color though. Let's do that now.

Uh-oh. We've run into a problem. As said before, we now have two rendered nodes. One that sets the fill color, and one that draws the rectangle. In which order do we execute them? How do we specify this order?

Let's work around the problem. We'll give the draw rectangle node a "fill" attribute. Then, inside of the node, we call the Processing "fill" command first, and than the "rect" command.

We know set only one fill color. Remember, from our code, that the fill color changes depending on if the mouse was pressed or not. This means we'll need two colors, and some way to choose between the two.

The choice node does the trick. It checks a boolean value, "condition", which is connected to the mouse pressed port. If the condition is true, its output is set to input A. Otherwise, if the condition is false, we'll set the output to input B.

By having the choice determine the color, we can now use this color as the fill for our rectangle node.

This whole "fill" port feels like cheating though. What if there is more state we need to set? Will we need a huge list for each node that draws something? Imagine that the state itself is order-dependent: setting a rotation before a translation has a different effect than the other way around. In which order do we set the state inside of the node itself?

Clearly, if we want to fully support the stateful nature of Processing, we need a different solution.

Quartz Composer does this by tagging each node with an order number. Instead of one rendered node, we now have a list of nodes that needs to be executed in the specified order. Without the "mouse pressed" condition, the graph can look as follows:

By specifying the order, we make state ordering explicit. In this graph, the fill node is executed first, setting the global state fill color. After that, the "draw rectangle" node is evaluated, which processes its dependencies (the mouse) first after drawing something itself.

This seems like a good solution! By tagging the nodes with an order, we can choose the ordering.

How does this look when we add a condition? I don't know:

We can't execute both the black and white nodes, because they would set state and the last one would win. Also, we can't use a condition node since none of them have an output. They just set state.

One solution is to pass data around until the very end. In this example, we would choose between two color objects (that don't set state), then finally set state after we've chosen.

This works for this case. However, it is not sustainable for every kind of state.

The semantics of the network need to change to adapt this imperative way of handling code. In a way, it becomes a wrapper around the constructs we use in programming: conditions, loops, functions, etc. Figuring out these semantics will be part of my research project.

PyCon 2009 report

2009-04-15T00:00:00Z

This year was my first visit to PyCon in the United States. Here's the full schedule for the conference.

In this post, I will give some notes and remarks about the sessions that I followed. For each session I included a link to the session details. Most of them have videos and slides of the talks.

How to Give a Python talk

A lot of talk about presentations using slides. I don't like slides. The author didn't like live coding because you could make a lot of typos, which would frustrate an audience. Later on, somebody gave a lightning talk about player piano, a project that types automatically for you and is very useful for presentations.

» How to Give a Python talk

Introduction to CherryPy

CherryPy is a lightweight HTTP framework. It is very much unlike Django: it doesn't have an ORM or administrative interface, but just provides the perfect balance for creating web applications that are different from the standard CRUD.

You can install "tools" that can, for example, automatically encode/decode JSON. CherryPy-guy gave a nice example that I can't find, but here's the webpage.
You can use other dispatchers than the default. MethodDispatcher looks interesting, because it allows you to do REST-style systems.
There's a tool to serve static content.

» Introduction to CherryPy

Introduction to Python Profiling

Tools for how to write fast Python code.

You need to profile your code to find out where it spends its time. This is really important: you really don't know where your code is slow.
Use cProfile
KCacheGrind can help you with interpreting Python profiles.
RunSnakeRun allows you to view cProfile dumps in a GUI. This tool was created by the guy giving the talk.
Caching function references can speed up deeply nested code. But profile first.
Property lookups look just like an attribute lookup, but can be much slower.
Another interesting tool is line profiler, by Robert Kern of Enthought.

» Introduction to Python Profiling

Panel: Python VMs

A discussion about the different implementations of Python:

CPython, the "official" Python implementation. This gets confused with Python, because a few years ago there was only one Python language and implementation. Now there are many implementations of the same language.
Jython, Python on the JVM.
IronPython, Python on .NET and Mono.
PyPy, Python in Python. This is actually more useful than it sounds.
Unladen swallow, an effort by Google to make Python 10x faster.

What I found interesting in the discussion was that the "official" Python guys really welcome and support the other implementations. I had a chance to have lunch together with all the different VM guys, and they are really sincere about this.

Just as with JavaScript implementations, there is a lot of buzz and activity around making Python run faster. Python and JavaScript are very different languages however, and optimization techniques used in JavaScript cannot be used in Python. Still, everybody has their own idea of how to make Python faster (check out "PyPy status").

» Panel: Python VMs

Python in a sandbox

A talk by the makers of PyPy about the use of PyPy as a virtualization tool. PyPy has a secure sandbox for running untrusted Python code, where you can bound IO calls, CPU and RAM resources. Really interesting for running other peoples' Python code on your own server. PyPy is "only" 2-3 times slower than regular Python.

There is no official release, but here are the steps for running it yourself:

Download the sandbox.
Translate pypy with --sandbox flag (this takes a while).
Run the script using pypy interact.py

» Python in a sandbox

A better Python for the JVM

A very technical talk about the Jython compiler. A bit over my head. However the conclusions were:

Currently Jython is a bit slower than Python.
The new Jython compiler should make things a bit faster.
However, currently the new compiler isn't faster.
But it provides better opportunities for optimization.

» A better Python for the JVM

Behind the scenes of EveryBlock.com

Adrian Holovaty, maker of Django, talked about Everyblock, his project to catalog everything that's happening at the hyper-local level (in your neighborhood).

They have a lot of different data-types, and wanted to store all of them in the same table. This meant that date types had to be opaque, since the columns would be different for the various datatypes. The main table has columns named varchar1, varchar2, int1, int2, etc. A schema table defines what these various columns means for a given schema.

This really looked like a hack, which was surprising coming from Adrian, someone who has written such a clean web framework. It seems like an other database system (such as key-value stores or even a graph database, such as neo4j) would fit the job better.

One remark also struck me as odd: Adrian stated that he wrote everything from scratch, basically because "I don't trust other people's code". It was unclear whether he was entirely serious or not, but the fact remains that no external code was used in his website.

» Behind the scenes of EveryBlock.com

Jython Progress

The reason for coming to PyCon: a very interesting talk by Frank Wierzbicki, who works at Sun on Jython.

The focus for the Jython 2.5 release was on compatibility. Through our use with Jython we found that a lot of stuff that worked in Python really worked in Jython as well, which was nice.

The focus for Jython 2.6 will be on performance and integration with Java. It should show up quicker than the 2.2 > 2.5 release, which took years to complete.

At the end he also showed Field, which is a Processing/NodeBox-like environment in Java using Jython. It has only very recently been open-sourced. It only runs on Mac OS X 10.5.

Jython Progress

Pinax: a platform for rapidly developing websites

Pinax has been getting some exposure lately because it provides ready-made components for Django. I had some troubles getting it to work, and was hoping this talk would provide some answers.

There seemed to be some problems with how to distribute this, and how to version all of it. They make a lot of use of svn:externals, which isn't bad per se. It really requires a lot of commitment to get started with the framework, but once you have it installed, it provides a lot of the boilerplate functionality of sites, and more specifically, social networks: user-to-user messaging, twitter clone, tagging, photo management, interest groups, … .

After the talk, I'm still not sure if it's worth the trouble, or if I'd rather be writing a lot of that from scratch. I feel the framework is still a bit too young to be used without to much configuration hassle. Also, the remark of Adrian during his EveryBlock talk about writing everything himself rings true here.

» Pinax: a platform for rapidly developing websites

Class Decorators: Radically Simple

A nice introduction to "better metaclasses". The functionality is only available in Python 2.6, but the talk is interesting (and short) enough to watch in its entirety.

» Class Decorators: Radically Simple

PyPy status talk

After the talk about Python sandboxing using PyPy, I was interested in what the status of PyPy was as a project. PyPy was started to be able to generate Python interpreters with more flexibility than a fixed C implementation. By defining a Python Interpreter on a higher level, you can experiment with different VM features quickly.

The PyPy interpreter is slower but consumes a lot less memory: some objects are 50% the size of CPython. They don't support libraries written in C, but they do support CTypes as the official way to have bindings for PyPy.

» PyPy status talk

Drop ACID and think about data

Bob Ippolito talked about the various alternative data storage implementations that have come up recently. He covered both closed- and open-source implementations. Most of the talk was about alternative key-value stores and other non-relational models, but he didn't cover graph databases.

There are a lot of different kinds of databases out there, but none of them seemed particularly stable or usable for my purposes. Afterwards, I checked out the open space about Cassandra, which seemed like the most stable one.

» Drop ACID and think about data

Concurrency and Distributed Computing with Python Today

This talk was not so much about the multiprocessing module in Python 2.6, but more about the tools available today.

He covered Jython as a solution for the global interpreter lock (GIL), because Jython uses Java threading which does not have the GIL. Also, the usage of java.util.concurrent solves a lot of problems. Actually, this seemed the most interesting approach for my projects.

He also talked about Stackless Python, which offers lightweight threads and cooperative multitasking.

There are different approaches to multitasking:

Real threads: hard to use because concurrency requires a lot of thinking ahead. They are hampered by the GIL in CPython.
Coroutines: light-weight threads are not truly parallel, but simplify the threading problem.
Actors: isolated, self reliant components that communicate via messages. They are truly parallel, and are a good model to overcome the threading fallacies. Erlang and Scala are two programming languages that use the actors idiom for concurrency.

All of these approaches have libraries implementing them in Python. There are a lot of them, and most are alpha quality. The two good ones are Twisted and Kamaelia. The rest is a mish-mash of technologies. ("Concurrency is hard, let’s go shopping!")

» Concurrency and Distributed Computing with Python Today

Abstraction as Leverage

An abstract talk about the tower of abstractions we use every as programmers, and how to produce and consume abstraction layers.

The starting point for the talk was that abstractions "leak". This is something Joe Spolsky has written about, and an interesting read. He also pointed out a quote from Jason Fried of 37 signals: "Prefer action over abstr-action".

It's hard to sum up the talk without paraphrasing it (badly), so I suggest you go see it.

» Abstraction as Leverage

Paver: easy build and deployment automation for Python projects

Paver is sort of like Ant for Python projects, but instead of using a bizarre XML

syntax, just using Python. It is also very similar to Ruby's Rake. It is really

interesting and looks mature. It requires you to make a pavement.py file in your project

directory that gets picked up and executed by Paver.

The basic building blocks in Paver are tasks. Tasks can be given command options (using optparse -like syntax).

Paver can be used for deploying servers, generating documentation, building a distribution

(using distutils/setuptools), working with files, etc. Paver doesn't replace distutils, but

embraces and extends it in a non-evil way.

A good introduction is Getting Starting with Paver.

» Paver: easy build and deployment automation for Python projects

Making games in Python - Tools and techniques at CCP

I wanted to see this talk to see how a gaming company used Python almost everywhere in a high-profile MMORPG. They use Python for networking, the web server, content authoring tools (using wxPython) and game logic. Some performance-critical parts were they use C++. They did not talk about how they wrapped C++ code, only that they didn't use Boost.Python.

They use embedded stackless Python, which they actively maintain. It allows them to do micro-threading (every object in the world runs in its own light-weight thread).

They use their own code-reloading system called livecoding because the reload built-in was deemed unsuitable. Their custom code reloading system allows them to reload game code while server and client are running. They also integrated unit testing so that the tests run before the code is reloaded to see if it passes.

» Making games in Python - Tools and techniques at CCP

Stackless python in EVE, pt. 2

This talk went more in depth about how they used Stackless Python in EVE. It covered

StacklessIO, a unified framework for blocking operations in Stackless Python. He also talked

about how they deployed StacklessIO at CCP, using "Cowboy mode", which short- circuited normal

QA procedures. Although interesting, I could not see myself using any of the technologies or

working methods introduced.

» Stackless python in EVE, pt. 2

Designing a web framework: Django's design decisions

I love to see Jacob Kaplan-Moss give a talk, and this one was no exception. This talk was high-level overview of some of the design decisions they made in the framework, and how to keep ego out of the equation.

He talked about the difference between academic frameworks ("architecture astronauts") and the real world.

What I found interesting was the decision in Django to make a full stack framework instead of glueing together existing components. Part of the reason for this was that there were not much existing components available, but also that they could provide one consistent API

dialect that felt they same, whether you were working in the templating language, the database API or some other part of the stack.

He also pointed that out that having users pick from existing components because it would allow for full flexibility only makes sense for expert users, and not for beginners that are just starting out with your framework. Having them pick from a list of components that they have no knowledge over is silly.

He also pointed out the talk by Cal Henderson of Flickr, Why I Hate Django which is really interesting and funny.

» Designing a web framework: Django's design decisions

Open Space: Cassandra

I followed an open space talk by Jonathan Ellis of Rackspace Managed Hosting, one of the maintainers of Cassandra. [Cassandra](http://code.google.com/p/the- cassandra-project/) is a distributed database somewhere between BigTable and Dynamo, which are both closed source. The project was started by two ex-Googlers at Facebook, and has now been open-sourced. The talk went into depth about the implementation of Cassandra.

The system looks interesting but way too big for my needs. As long as we don't really need high-performance distributed databases, I'd prefer my ACID using MySQL or Postgresql (or Neo4j).

Neo4j

Turns out the maker of the new Jython compiler also works on an open-source graph database called neo4j. His name is Tobias Ivarsson. I talked to him and showed him Perception. He was convinced that neo4j would be perfect for this: super-fast, O(1) lookups, lightweight, Python bindings. It is a mature graph database that runs in production for over 5 years.

There are components available for indexing, graph algorithms and much more.

Answered questions

I came to the conference to hear talks about the progress and future of Jython. I am now convinced that Jython is an excellent choice for Python development, given that there are some very motivated people working on it, and they have the full support of the CPython guys.

Unanswered questions

One thing I've been struggling with in NodeBox is how to do proper packaging of NodeBox packages. I need a system that can version packages, do dependency management, can run several version of the same package at the same time, and do live loading/unloading of packages. I recently discovered OSGi recently, which seems to solve this problem for Java. However, Python doesn't seem to have a system like that available. There was a lot of talk about virtualenv, pip and even zc.buildout, but none of those seems to provide a complete answer. (By the way, Jacob's recent blog post about zc.buildout is very interesting.) The whole environment seems very much in flux, and I think we'll have to wait for the next PyCon to have an answer available.

Personally, I would love to see integration between the VM's native package management system and the Python implementation (for Jython, that would be OSGi). However, even OSGi seems to come under attack, as Sun is [rolling its own solution](http://www.osgi.org/blog/2007/07/can-someone-tell-sun-about- osgi.html).

Overall, package management is a hairy business, and one that needs some serious thought. As for now, I think I'm better off taking a good hard look at OSGi and then rolling my own solution.

My Favourite Python hack

2009-03-27T00:00:00Z

In NodeBox 1 we have this cool feature that we call "the throttle". It allows you to interactively drag any number in the (Python) script and see the results immediately. Whether it's and X/Y coordinate, the number in "for i in range(10)", or a number in a complicated formula, just dragging the number helps you visualize the effect of it on the overall composition. Here's a movie that demonstrates how this works:

Today I'd like to talk about the implementation, which works like this:

Find the character under the cursor.
Expand until you get the full number (include digits, periods and minus signs)
Replace it with magic_var and add this variable to the locals dictionary
When the user "drags" the number, replace the value of magic_var with an updated value and run the script again.
When the user releases the mouse, replace the old number in the source file with the new number.

Sounds simple, right? While this approach worked well in 90% of the cases, it broke down when the signs of the number changes from positive to negative or vice versa. Here are the three use cases:

If an addition goes from positive to negative the sign should change:
```
random(alpha+8) -> random(alpha-8)
```
If a subtraction goes from negative to positive the sign should change as well:
```
random(alpha-8) -> random(alpha+8)
```
But if a negative number turns positive, the sign should disappear:
```
random(-8) -> random(8)
```

The difficulty was in finding the difference between the first case and the third case. After a lot of dabbling around with regular expressions, I suddenly had an insight. There is actually a Python module that handles this problem: the Python parser itself.

When Python parses the code, it builds an abstract syntax tree (AST), a tree-like representation of the code. Different nodes are represented by classes in the compiler.ast module. The use cases above are represented by instance of Sub (a two-part subtraction), UnarySub (a negative number), and Addition (a two-part addition).

Note that we replaced the original number with the name magic_var in our original source code, and compiled that. After compilation succeeds and returns an AST, a function called checkSigns recursively checks the AST for special sign cases. First, it tries to find the magic variable. Once it has found that, it checks the parent node to see if it encounters any of the special cases, and basically prepares the object for string replacement afterwards.

What I like about this hack is that if it works correctly, you don't even notice it's there.

If you'd like to view the code, it's in the NodeBox source as ValueLadder.py.