What does Bonsai, the Japanese art form of carefully shaping miniature trees have to do with scientific research? For Gonçalo Lopes, a graduate student at the Champalimaud Centre for the Unknown, Bonsai perfectly captures the essence of his doctoral thesis – “just like the art form, the main aesthetics and challenge of our work involve being able to recreate all the complexities of a living landscape in a very confined space.”
How a side project became a powerful open-source tool.
The living landscape Gonçalo is referring to is a complex naturalistic setting, where animals experience a rich environment and face novel challenges. “In my project”, he recalls, “I sought out to discover how the brain uses prior knowledge about the way the world works, acquired in a stimulating and diverse surrounding, to solve new problems.”
Figure 1. On the top examples of processing pipelines for tracking the activity of two colored objects in a video stream using Bonsai. IN the featured image you can see the Japanese art of recreating complex landscapes in a compact and elegant form.
In other words, “we wanted to know how the brain understands the world. When we look at objects like a cup, or a ball, we find it natural that we know what to do with them. We can imagine how the object would feel like if we picked it up, or sound like if we threw it around, and we use this information to guide our decisions and overcome obstacles. In order to learn more about how brains can actually build and use this knowledge, we decided to observe animals interacting with rich environments where they have to figure out how to tackle new situations using their past experience.”
“In addition to carefully monitoring behavior, we also wanted to take control over these environments in order to define their rules of operation – the physics and statistics that the animal would have to learn about.” – Adam Kampff, principal investigator at the Champalimaud Centre for the Unknown.
When trying to tackle such a complex problem, it is crucial to record extensive measures of the animal’s behavior with high-temporal and high-spatial resolution, often from multiple sensors in parallel. Adam Kampff, one of Gonçalo’s thesis advisors, together with Joe Paton explains, “in addition to carefully monitoring behavior, we also wanted to take control over these environments in order to define their rules of operation – the physics and statistics that the animal would have to learn about. These two problems of measurement and control require a lot of experimentation with many different state-of-the-art hardware and software solutions. You don’t know a priori which one will work, so you have to try many of them.”
video 1 This movie shows Bonsai being used to move metal rods connected to motors, according to the proximity of an object (in this case a hand-held rod).
While approaching his research question, Gonçalo found that the optimal way of testing it was difficult to pin down. “I set up multiple different environments, in the search for the most suitable one, a process that was very time consuming. As I was transitioning between different setups I realized that I needed better tools that would allow me to test many different ideas more quickly.” That is how Gonçalo came up with Bonsai, a programming language that enables the user to efficiently measure and control multiple variables within an experimental setup.
“Just like the art form, the main aesthetics and challenge of our work involve being able to recreate all the complexities of a living landscape in a very confined space.” – Gonçalo Lopes, graduate student at the Champalimaud Centre for the Unknown.
“Bonsai provides the user with many powerful building blocks and the means to combine them into complex designs using very compact specifications. Instead of going through the regular process of spending many hours and days writing code to try out all different hardware and software solutions, while not knowing which one will actually work, I ended up developing Bonsai – a tool where I could quickly play around with all the different solutions without committing myself too deeply to any one of them”.
Video 2. In this video Bonsai is controlling the physics of digital balls that are being projected on a white-board. Bonsai analyses the lines drawn on the board and guides the path and movement of the balls to follow the lines.
As it turned out, Gonçalo still had to spend many days and nights programming in order to create Bonsai. However, “in the end, not only did I get to try out all the different solutions I had in mind, but the tool itself kept on expanding and growing and giving us new ideas of things to try. Like a good playground, a good tool also primes your imagination for what is possible, and that was a very satisfying consequence of developing Bonsai.”
As many projects developed at the Champalimaud Neuroscience Programme (CNP), the full scope of Bonsai is the result of a collaborative effort with other CNP members. As Gonçalo recalls, “In the beginning, I started developing Bonsai for myself, mostly to support the research project I was working on. However, early on I started showing it to people around the CNP. Niccolò Bonacchi was the first to get involved and together we brainstormed the design of many aspects of the user interface that would make Bonsai accessible to even more people. Nico has a keen eye for how user interfaces will be used and provided many valuable insights into making the graphical environment as consistent as possible.”
Figure 2. From the left: Gonçalo Lopes, Niccolò Bonacchi and João Frazão. Photo by Gabriela Martins.
Later on, João Frazão joined the team. Before joining the CNP, Gonçalo and João worked together for four years at YDreams, designing a framework for rapid prototyping of augmented reality applications and a lot of the expertise acquired during that period has been applied to the development of Bonsai. “João is an amazingly creative software engineer”, says Gonçalo, “he helped polishing many of the conceptual edges of the language to make sure it would be as general and powerful as possible.”
“One of my favourite applications of Bonsai was the development of a crowd version of the “Pong” videogame… It was quite a spectacular sight.” – Gonçalo Lopes.
Indeed, once created, Bonsai was quickly picked up by many users at the CNP. “The amazing part of working on a shared tool is that your users, especially the early adopters, end up thinking about the problem together with you. A lot of time was spent debating aspects of the language with power-users like Bassam Atallah, that were already experienced in using many other neuroscience tools, and as such, knew a lot about what did and did not work and about what features would be the most useful.”
“From these particular constraints and requirements I was trying to extract the set of features that would be the most general. Solving not only specific scenarios, but enabling the expression of a large variety of experimental designs. All the authors in the Bonsai paper (1) helped to push the boundaries of what could be done in Bonsai by developing state-of-the-art neuroscience experiments that are now used to demonstrate all the different things you can do with the language. I am still impressed by how much could be achieved in such a short period of time!”
Currently, Bonsai is being used not only by CNP members, but also by scientists in other institutions. “Researchers at the CNP have been amazing early adopters of the technology, with 12 out of the 17 labs using it actively, for a total of more than 50 users.” Outside the CNP, and following the public release of Bonsai, it is now being used at several neuroscience institutes, including labs at University College London, Harvard University, Brown University, Cold-Spring Harbor Laboratories, Donders Institute for Brain, Cognition and Behavior and many others. “It is now also being taught at experimental neuroscience courses such as the Transylvanian Experimental Neuroscience Summer School and the CAJAL Advanced Neuroscience Training Programme” adds Gonçalo.
Figure 3. Bonsai being used to play an interactive game of Pong by an audience of 400 people during the Ar|Respire Connosco event on Emergence. On the top, Gonçalo Lopes (left) and Adam Kampff. On the bottom, audience members participating in the game. Photos by Wieland Brendel.
The uses of Bonsai, however, are not restricted to the lab. “One of my favorite applications of Bonsai was the development of a crowd version of the “Pong” videogame”, Gonçalo recalls. “In this variation, we were able to get an entire auditorium of 400 people to play the game.” The game was played between two teams – the right side of the audience was controlling the right Pong paddle and vice-versa. Each person was holding a card that had two sides, a red one and a blue one. As the ball was moving towards the right, each member had to hold up the red side of the card facing the stage if they wanted the paddle to go up, or the blue if they wanted the paddle to go down.
“We used Bonsai to capture images with a color camera and process them online with a color segmentation filter in order to measure the amount of blue and red in the image as the audience raised the cards in the air. We then used the ratio between blue and red to determine how high the paddle was on the screen. Then we simply added a second camera and we were able to have two teams in the auditorium playing Pong against each other. It was quite a spectacular sight.”
“I’m happiest when someone who does not have a formal programming background is actually able to create a piece of state-of-the-art reactive software using Bonsai that would blow the mind of many professional programmers.” – Gonçalo Lopes.
Recently, the team has been focusing on making Bonsai even more accessible and easy to use. “As more and more users were interested in using it for their own research, the need to make the interface much more intuitive and self-explanatory became a central concern, and a lot of effort was spent in trying to make every command explicit and in providing better ways to guide new users. We also created a wiki, tutorials and a forum for users to allow the community to share their own experience and feedback of using the language. Our ultimate goal is that anyone will be able to use it and we are trying to provide the means to make the initial learning curve as smooth as possible.”
Gonçalo describes his main goal as empowering users to be able to tell their computers to do progressively more complicated things with ease. “A direct consequence of this, I believe, will be mind-blowing science, creative digital art and new ways of interacting and playing with one of the most amazing tools ever created by humanity, the digital computer. I’m happiest when someone who does not have a formal programming background is actually able to create a piece of state-of-the-art reactive software using Bonsai that would blow the mind of many professional programmers.”
“Finally, I really need to thank my supervisors Adam Kampff and Joe Paton for being so supportive and encouraging of this project from the beginning. The specific goal of my PhD was not to develop Bonsai but it was very encouraging to have them realize the potential of the project and let it grow to fruition. Adam now even uses it extensively for teaching, has become one of its most proficient users and is a main driver for the adoption of Bonsai outside the CNP.”
FOR THE ORIGINAL STUDY SEE: Lopes G, et al.(2015) Bonsai: An event-based framework for processing and controlling data streams. Neuroinform. 9 (7).
Liad Hollender works at the Science Communication office at the Champalimaud Neuroscience Programme.
Edited by: Ivo Marcelo (section editor), Clara Howcroft Ferreira (editor-in-chief)
Loading Likes...
You must be logged in to post a comment.