This guide is broken up into three parts:
BehaviorSpace is a software tool integrated with NetLogo that allows you to perform experiments with models. It systematically varying the values of sliders and records the results of each corresponding model run. This way you can explore the model’s “space” of possible behaviors and determine which combinations of slider values cause the behaviors of interest.
The need for this type of experiment is revealed by the following observations: Models often have more than one slider, each of which can take a large range of values. Together they form what in mathematics is called a parameter space for the model, whose dimensions are the number of sliders, and in which every point is a particular combination of values. Running a model with different slider values (and sometimes even the same ones) can lead to drastically different behavior in the system being modeled. So, how are you to know which particular configuration of values, or types of configurations, will yield the kind of behavior you are interested in? This amounts to the question of where in its huge, multi-dimension parameter space does your model perform best?
Suppose you want speedy synchronization from the agents in the Fireflies model. The model has four sliders — number, cycle-length, flash-length and number-flashes — that have approximately 2000, 100, 10 and 3 possible values, respectively. That means there are 2000 * 100 * 10 * 3 = 600,000 possible combinations of slider values! Trying combinations one at a time is hardly an efficient way to learn which one will evoke the speediest synchronization.
BehaviorSpace offers you a much better way to solve this problem. If you specify a subset of values from the ranges of each slider, it will run the model with each possible combination of those values and, during each model run, record the results. In doing so, it samples the model’s parameter space — not exhaustively, but enough so that you will be able to see relationships form between different sliders and the behavior of the system. Once all the runs are over, you are presented with a visual display with which you can explore the data from all the runs and determine how well each exhibited the behavior you are investigating.
The idea behind BehaviorSpace is that the way to truly understand a model is to run it multiple times with different parameter (slider) settings in order to see the whole range of behaviors the system is capable of producing. Only then is it possible to investigate when and why certain behaviors arise. Isolated trials are insufficient for this purpose because you have no reason to assume that the model will always demonstrate the particular behaviors you see. It’s like eating in one restaurant in New York and then claiming you’ve seen all that the city has to offer. By enabling you to explore the entire parameter space of a model, and thus its entire space of behaviors, BehaviorSpace can be a powerful tool for model understanding.
You begin using BehaviorSpace by setting up an experiment. Its purpose is to run the model many times and collect data to determine which run best exhibits the behavior you are after. In the case of Fireflies, as mentioned above, data might be collected to reveal how quickly synchronization happened in each run.
Setting up an experiment initially entails specifying in the slider values dialog what possible values each slider in the model should take in the experiment, either in the form of a range or an arbitrary list, which is then displayed in the sliders list.
You can also dictate how long each run should last, either by entering the number of times the forever button should be called in the time clicks text-box, or by providing a reporter in the exit condition text-area that causes the run to stop when it reports a boolean value of true.
Next, you must specify a behavior reporter, which is the data you want continually recorded during a model run. For Fireflies, this would be the current extent of synchronization in the model, say the number of fireflies currently lit up, which you could enter with the following reporter:
count turtles with [color = yellow]
By collecting this data repeatedly throughout a run, an overall assessment can later be made as to how quickly synchronization was achieved.
Finally, you may select a setup button to initialize the model and the forever button that continues it.
By inputting the above information, you have told the system enough that it can run an experiment. As the experiment proceeds, it will produce data about each model run based on the behavior reporter you gave it.
In its raw form, this data is dry and difficult to interpret — it is just long series of numbers. BehaviorSpace’s job makes the data intuitively accessible by presenting it visually. It does so by allowing you to hold the values of all sliders but two constant, then presenting you with a grid of squares, called a fitness landscape.
The axes of the fitness landscape represent the two selected sliders. Each square in the landscape then represents the particular run whose slider values are determined by the location of the square along the two axis and the current values of the sliders held constant.
Example: Suppose you are performing an experiment with the Fireflies model that tried many possible combinations of slider values. Then you select flash-length and number-flashes as the sliders to represent the axes of the landscape, with number held constant at 1500 and cycle-length at 100. Assuming flash-length is on the horizontal axis and number-flashes is on the vertical axis, the landscape would be 10 by 3, since flash-length goes from 1 to 10 and number-flashes from 1 to 3. Each square on the landscape would be colored a particular shade of blue, ranging from solid blue to black, depending on how good (i.e. “fit”) the behavior of the run represented by that square was according to criteria you select. For instance, squares representing runs of Fireflies in which the agents synchronized their lights quickly would be more blue than those representing runs where the fireflies synchronized slowly or not at all.
The reason the grid is called the “fitness landscape” is because it displays a 2-dimensional landscape of squares, each with a fitness, allowing you to take in with a glance regions of high and low fitness.
The criteria with which you rank run data — and thereby determine fitness — can be based on simple point statistics, such as maximum or mean value, or by slope, which is sensitive to the data’s behavior (not to be confused with the model’s behavior) over time.
In addition to changing the values of the constant sliders and alternating which sliders get to be on the axes of the landscape (these two operations let you explore the model’s parameter space), you can see the data from each run represented on the landscape plotted on a behavior plot by moving your mouse over that run’s corresponding square.
You can learn more about how evaluation of the experimental data works in the evaluation window part of the Interface Reference section below.
This section describes the interface to BehaviorSpace. Included is a general description of each of the three main BehaviorSpace windows, a list of the interface components they contain, as well as a description of those components. It is intended mainly as a reference. The windows are discussed in the order you use them: first the Setup Window, then the Progress Dialog, and finally the Evaluation Window.
This is the window in which you set up an experiment. Setting up an experiment requires entering information in the fields outlined below, then pushing the run comparison button. When opening and closing models, or changing the existing model, the setup window is automatically updated, so there is no need to close and reopen it.
Sliders list: The current list of sliders in the model, which is updated automatically as sliders are added, changed or deleted. The values specified to the right of each slider are the current values that slider is assigned to take in the experiment.
Change values button: Brings up the Slider values dialog so you can change slider values.
Slider values dialog: The dialog that pops up when the change values button is pushed.
Behavior reporter text area: A place to enter a NetLogo reporter to specify what kind of data is to be collected during each model run. This reporter is evaluated before every time click, or call to the forever button, and once at the end of the run, and its value is recorded for later analysis.
Example: If you are interested in the population of sick people in the Virus model, you might want your behavior reporter to report how many sick people there currently are in the model with the following behavior reporter:count people with [sick?]Or, if you are investigating the distance between each turtle and the center of the screen in Turtles Circling, you might have it report the mean of all the turtle’s distances to the center:mean values-from turtles [distancexy 0 0]
Time clicks checkbox and text-field: When the checkbox is selected, each model run will end after the number of time clicks specified in the text-field. (One time click means one push, or iteration, of the forever button.)
Exit condition checkbox and text-field: When the checkbox is selected, each model run will continue until the reporter entered in the text-field reports true. This reporter is run after every time click. The exit condition can be any valid NetLogo reporter that evaluates to true or false, such as those that use the operators =, <, >, <=, or >=
Example: The following exit condition causes model runs of Wolf-Sheep Predation to stop after either population is extinct:(not any sheep) or (not any wolves)
Setup and forever button menus: Allow you to select the optional setup button and mandatory forever button for an experiment. The setup button, if selected, is called once at the beginning of a model run, while the forever button is called repeatedly to run the model.
Run comparison button: Begins the experiment, which is based on the input to the interface elements listed above.
Graphics window: Displays the state of the current model run.
Graphics on? checkbox: When unselected, graphics are temporarily turned off, although the model proceeds normally.
Current run number / slider values / time click text: Displays the number of the current run being performed, the values of the sliders in that run, and the number of time clicks that have elapsed
Behavior plot: Plots the value of the behavior reporter as the model run progresses.
Cancel button: Aborts the experiment, at which point the progress dialog quits.
It works by holding the values of all but two sliders constant, then drawing a grid of squares, called a fitness landscape, that represents all possible value combinations of the last two. The values held constant by the former sliders can be changed, and any of these sliders can be swapped in for one of the two sliders representing the axes on the grid. In this way you are essentially focusing on two sliders at a time, seeing what relationships exist between their values and the behavior of the model.
At the top of the window is a collection of radio boxes and pull-down menus, through which you can select criteria to rank the data reported by the different runs. These criteria can be changed as often as you wish, enabling you to highlight more than just one property of the data. Data can be ranked by slope characteristics, such as how much it increases over time, or the extent to which it can be fit to an exponential curve; or, the data can be ranked by a point statistic, such as its mean or minimum value.
Finally, you are able to mouse-over the landscape, and as the mouse passes over each square, which represents a single model run, the data from that run is plotted on the behavior plot to the left. Every time an experiment is run, a new evaluation window is created, so there can potentially be many on the screen at once. Each, however, only represents the data from the experiment that created it, so even if you change the name or number of sliders in your model, existing windows will remain unchanged.
Point statistic type menu: Determines whether run data should be ranked according to their minimum, maximum, mean, or last value, or by the number of time clicks they lasted for. The latter option is only relevant when an exit condition was entered to stop runs, otherwise they will all finish in the same number of time clicks.
Point statistic value menu: Determines whether it is “better” for the statistic selected in the point statistic type menu to be high, low, or near 0.
Slope type menu: Controls whether the run data is fit to a line or exponential curve. This distinction is only relevant with regards to error, since the slope of the data always remains the same. The higher the weight selected in the relative weight menu, the less this menu selection matters, because that means slope matters increasingly more than error. Error is determined by attempting to fit the data to either a line or a curve, depending on which is chosen, and then finding the Least-Square Error involved in doing so.
Slope value menu: Determines whether it is “better” for the data to be increasing (the more positive the slope the better), decreasing (the more negative the better), or constant (the closer to 0 the better)
Relative weight menu: Determines the relative importance of the slope of the data compared to how well it fits a line or exponential curve (see slope type menu). If 60 is chosen, then 60% of the data’s rank comes from its slope (how much it is increasing, decreasing or constant) and 40% comes from line and curve-fitting error. So, a value of 100 means only slope is considered, while a value or 0 means only the error is considered.
Fitness landscape: Consists of a grid of squares, each of which represents the model run whose slider values were the current values of the constant sliders below and the ith and jth values of the two sliders on the grid’s axes, where ‘i’ is the horizontal index of the square and ‘j’ is the vertical. The entire grid thus represents all runs in which the constant sliders had their current values (the current values being the ones displayed on the set of slider components held constant below the grid). This grid earns the name fitness landscape because the squares of which it is composed each has a fitness value, expressed by the color of the square, enabling you to see a landscape of colors denoting regions of high and low fitness — that is, sets of neighboring near-black and near-blue squares. The landscape represents a 2-dimensional plane in the multi-dimensional parameter space of the model, since it accounts for all possible values of two sliders, while the rest are held constant. By changing the values of any constant sliders you are shifting that 2-D plane through the parameter space, and by selecting new sliders to be the axes, you are selecting a new plane entirely (that is, a plane through different dimensions).
Set of sliders components held constant: These sliders consist of all those not present as axes of the fitness landscape. Since all the model runs represented on the landscape have the same value for these sliders — namely, the one currently selected on the slider components — they are considered held constant for the landscape. By changing the value of a single one of these sliders the whole landscape changes, because it now represents a whole new set of runs. If you changed a slider from 10 to 12, say, then all the runs in which that slider had the value 10 are replaced by those in which it had the value 12.
Behavior plot: Displays the data from individual runs as you move the mouse over their corresponding squares on the fitness landscape. It’s x-axis is time clicks and its y-axis is behavior — that is, the value reported by the behavior reporter entered in the setup window.
Add best-fit-line checkbox: Adds the line or exponential curve that best fits the data to the behavior plot, depending on whether ‘linear’ or ‘exponential’ was selected in the slope type menu.
Superimpose plots checkbox: Stops old data from being removed from the behavior plot when new data is added, resulting in increasing sets of data being superimposed on each other. Unselected, the checkbox causes all data to be removed.
Export Plot button: Saves the data currently being displayed in the behavior plot to a file.
Export Behavior Data button: Saves the behavior data gathered during all of the model runs to a file. The data saved for each model run includes: the slider settings for that run; how many time ticks the run lasted; the minimum, maximum, average, and final value for the behavior reporter; and the value of the behavior reporter at each time tick during the run. (Note that the fitnesses are not exported, only the raw behavior data. In a future version of BehaviorSpace the fitness data will be exported as well.)
Note: The export buttons create files in plain-text, “comma-separated values” (.csv) format. CSV files can be read by most popular spreadsheet and database programs as well as any text editor.