This guide has three parts:
BehaviorSpace is a software tool integrated with NetLogo that allows you to perform experiments with models. It runs a model many times, systematically varying the model's settings and recording the results of each model run. This process is sometimes called "parameter sweeping". It lets you explore the model's "space" of possible behaviors and determine which combinations of settings cause the behaviors of interest.
The need for this type of experiment is revealed by the following observations. Models often have many settings, each of which can take a range of values. Together they form what in mathematics is called a parameter space for the model, whose dimensions are the number of settings, and in which every point is a particular combination of values. Running a model with different settings (and sometimes even the same ones) can lead to drastically different behavior in the system being modeled. So, how are you to know which particular configuration of values, or types of configurations, will yield the kind of behavior you are interested in? This amounts to the question of where in its huge, multi-dimension parameter space does your model perform best?
For example, suppose you want speedy synchronization from the agents in the Fireflies model. The model has four sliders -- number, cycle-length, flash-length and number-flashes -- that have approximately 2000, 100, 10 and 3 possible values, respectively. That means there are 2000 * 100 * 10 * 3 = 600,000 possible combinations of slider values! Trying combinations one at a time is hardly an efficient way to learn which one will evoke the speediest synchronization.
BehaviorSpace offers you a much better way to solve this problem. If you specify a subset of values from the ranges of each slider, it will run the model with each possible combination of those values and, during each model run, record the results. In doing so, it samples the model's parameter space -- not exhaustively, but enough so that you will be able to see relationships form between different sliders and the behavior of the system. After all the runs are over, a dataset is generated which you can open in a different tool, such as a spreadsheet, database, or scientific visualization application, and explore.
By enabling you to explore the entire "space" of behaviors a model can exhibit, BehaviorSpace can be a powerful assistant to the modeler.
Old versions of NetLogo (prior to 2.0) included an earlier version of the BehaviorSpace tool. That version was much different. It wasn't nearly as flexible in the kinds of experiments it let you set up. But, it had facilities for display and analyzing experiment results that are missing from the current version. With the current version, it is assumed that you will use other software to analyze your results. We hope to re-add data display and analysis facilities to a future version of BehaviorSpace.
To begin using BehaviorSpace, open your model, then choose the BehaviorSpace item on NetLogo's Tools menu.
The dialog that opens lets you create, edit, duplicate, delete, and run experiment setups. Experiments are listed by name and how by model runs the experiment will consist of.
Experiment setups are considered part of a NetLogo model and are saved as part of the model.
To create a new experiment setup, press the "New" button.
In the new dialog that appears, you can specify the following information. Note that you don't always need to specify everything; some parts can be left blank, or left with their default values, depending on your needs.
Experiment name: If you have multiple experiments, giving them different names will help you keep them straight.
Vary variables as follows: This is where you specify which settings you want varied, and what values you want them to take. Variables can include sliders, switches, choosers, and any global variables in your model.
Variables can also include screen-edge-x and screen-edge-y, and random-seed. These are not, strictly speaking, variables, but BehaviorSpace lets you vary them as if they were. Varying screen-edge-x/y lets you explore the effect of world size upon your model. Varying random-seed lets you repeat runs by using a known seed for the NetLogo random number generator. Note that you're also free to use the random-seed command in your experiment's setup commands. For more information on random seeds, see the Random Numbers section of the Programmer's Guide.
You may specify values either by listing the values you want used, or by specifying that you want to try every value within a given range. For example, to give a slider named number every value from 100 to 1000 in increments of 50, you would enter:
["number" [100 50 1000]]
Or, to give it only the values of 100, 200, 400, and 800, you would enter:
["number" 100 200 400 800]
Be careful with the brackets here. Note that there are fewer square brackets in the second example. Including or not including this extra set of brackets is how you tell BehaviorSpace whether you are listing individual values, or specifying a range.
Also note that the double quotes around the variable names are required.
You can vary as many settings as you want, including just one, or none at all. Any settings that you do not vary will retain their current values. Not varying any settings is useful if you just want to do many runs with the current settings.
What order you list the variables in determines what order the runs will be done in. All values for a later variable will be tried before moving to the next value for an earlier variable. So for example if you vary both x and y from 1 to 3, and x is listed first, then the order of model runs will be: x=1 y=1, x=1 y=2, x=1 y=3, x=2 y=1, and so on.
Repetitions: Sometimes the behavior of a model can vary a lot from run to run even if the settings don't change, if the model uses run numbers. If you want to run the model more than once at each combination of settings, enter a higher number here than one.
Measure runs using these reporters: This is where you specify what data you want to collect from each run. For example, if you wanted to record how the population of turtles rose and fell during each run, you would enter:
count turtles
You can enter one reporter, or several, or none at all. If you enter several, each reporter must be on a line by itself, for example:
count frogs
count mice
count birds
If you don't enter any reporters, the runs will still take place. This is useful if you want to record the results yourself your own way, such as with the export-world command.
Measure runs at every tick: Normally NetLogo will measure model runs at every tick, using the reporters you entered in the previous box. If you're doing very long model runs, you might not want all that data. Uncheck this box if you only want to measure each run after it ends.
Setup commands: These commands will be used to begin each model run. Typically, you will enter the name of a procedure that sets up the model, typically setup. But it is also possible to include other commands as well.
Go commands: These commands will be run over and over again to advance to the model to the next "tick". Typically, this will be the name of a procedure, such as go, but you may include any commands you like.
Stop condition: This lets you do model runs of varying length, ending each run when a certain condition becomes true. For example, suppose you wanted each run to last until there were no more turtles. Then you would enter:
not any? turtles
If you want the length of runs to all be of a fixed length, just leave this blank.
Final commands: These are any extra commands that you want run once, when the run ends. Usually this is left blank, but you might use it to call the export-world command or record the results of the run in some other way.
Time limit: This lets you set a fixed maximum length for each run. If you don't want to set any maximum, but want the length of the runs to be controlled by the stop condition instead, enter 0.
When you're done setting up your experiment, press the "OK" button, followed by the "Run" button.
You will be prompted to select the formats you would like the data from your experiment saved in. Data is collected for each interval, run or tick, according to the setting of Measure runs at every tick option.
Table format lists each interval in a row, with each metric in a separate column. Table data is written to the output file as each run completes. Table format is suitable for automated processing of the data, such as importing into a database or a statistics package.
Spreadsheet format calculates the min, mean, max, and final values for each metric, and then lists each interval in a row, with each metric in a separate column. Spreadsheet data is more human-readable than Table data, especially if imported into a spreadsheet application.
(Note however that spreadsheet data is not written to the results file until the experiment finishes. Since spreadsheet data is stored in memory until the experiment is done, very large experiments could run out of memory. And if anything interrupts the experiment, such as a runtime error, running out of memory, or a crash or power outage, no results will be written. For long experiments, you may want to use both spreadsheet and table formats so that if something happens you'll at least get a table of partial results.)
After selecting your output formats, BehaviorSpace will prompt you for the name of a file to save the results to. The default name ends in ".csv". You can change it to any name you want, but don't leave off the ".csv" part; that indicates the file is a Comma Separated Values (CSV) file. This is a plain-text data format that is readable by any text editor as well as by most popular spreadsheet and database programs.
A dialog will appear, titled "Running Experiment". In this dialog, you'll see a progress report of how many runs have been completed so far and how much time has passed. If you entered any reporters for measuring the runs, and if you left the "Measure runs at every tick" box checked, then you'll see a plot of how they vary over the course of each run.
You can also watch the runs in the main NetLogo window. (If the "Running Experiment" dialog is in the way, just move it to a different place on the screen.) The view and plots will update as the model runs. If you don't need to see them update, then use the checkboxes in the "Running Experiment" dialog to turn the updating off. This will make the experiment go faster.
If you want to stop your experiment before it's finished, press the "Abort" button. But note that you'll lose any results that were generated up to that point.
When all the runs have finished, the experiment is complete.
It is possible to run BehaviorSpace experiments "headless", that is, from the command line, without any graphical user interface (GUI). This is useful for automating runs on a single machine or a cluster of machines.
No Java programming is required. Experiment setups can be created in the GUI and then run later from the command line, or, if you prefer, you can create or edit experiment setups directly using XML.
It is easiest if you create your experiment setup ahead of time in the GUI, so it is saved as part of the model. To run an experiment setup saved in a model, here is an example command line:
java -Xmx512M -cp NetLogo.jar \ org.nlogo.headless.HeadlessWorkspace \ --model Fire.nlogo \ --experiment experiment1
After the named experiment has run, the results are sent to standard output in CSV format. If you want to send the results to a file instead, add the --results flag and a pathname, e.g.:
java -Xmx512M -cp NetLogo.jar \ org.nlogo.headless.HeadlessWorkspace \ --model Fire.nlogo \ --experiment experiment1 \ --results results.csv
When running the HeadlessWorkspace class as an application, it forces the system property java.awt.headless to be true. This tells Java to run in headless mode, allowing NetLogo to run on machines when a graphical display is not available.
Note the use of -Xmx to specify a maximum heap size of 512 megabytes. If you don't specify a maximum heap size, you will get your VM's default size, which may be unusably small. (512 megabytes is an arbitrary size which should be more than large enough for most models; you can specify a different limit if you want.)
The --model argument is used to specify the model file you want to open.
The --experiment argument is used to specify the name of the experiment you want to run. (At the time you create an experiment setup in the GUI, you assign it a name.)
Here's another example that shows some additional, optional arguments:
java -Xmx512M -cp NetLogo.jar \ org.nlogo.headless.HeadlessWorkspace \ --model Fire.nlogo \ --experiment experiment2 \ --screen-edge-x 100 \ --screen-edge-y 100 \ --no-results
Note the use of the optional --screen-edge-x and screen-edge-y arguments to specify a different world size than that saved in the model. (It's also possible for the experiment setup to specify values for screen-edge-x and screen-edge-y; if they are specified by the experiment setup, then there is no need to specify them on the command line.)
Note also the use of the optional --no-results argument to specify that no output is to be generated. This is useful if the experiment setup generates all the output you need by some other means, such as exporting world files or writing to a text file.
Yet another example:
java -Xmx512M -cp NetLogo.jar \ org.nlogo.headless.HeadlessWorkspace \ --model Fire.nlogo \ --experiment experiment2 \ --table table-output.csv \ --spreadsheet spreadsheet-output.csv
The optional --table <filename> argument specifies that output should be generated in a table format and written to the given file as CSV data. If - is specified as the filename, than the output is sent to the standard system output stream. Table data is written as it is generated, with each complete run.
The optional --spreadhseet <filename> argument specified that spreadsheet output should be generated and written to the given file as CSV data. If - is specified as the filename, than the output is sent to the standard system output stream. Spreadsheet data is not written out until all runs in the experiment are finished.
Note that it is legal to specify both --table and --spreadsheet, and if you do, both kinds of output file will be generated.
The default output behavior, when no output formats are specified, is to send table output to the system standard output stream.
Here is one final example that shows how to run an experiment setup which is stored in a separate XML file, instead of in the model file:
java -Xmx512M -cp NetLogo.jar \ org.nlogo.headless.HeadlessWorkspace \ --model Fire.nlogo \ --setup-file fire-setups.xml \ --experiment experiment3
If the XML file contains more than one experiment setup, it is necessary to use the --experiment argument to specify the name of the setup to use.
The next section has information on how to create standalone experiment setup files using XML.
We don't yet have detailed documentation on authoring experiment setups in XML, but if you already have some familiarity with XML, then the following pointers may be enough to get you started.
The structure of BehaviorSpace experiment setups in XML is determined by a Document Type Definition (DTD) file. The DTD is stored in NetLogo.jar, as system/behaviorspace.dtd. (JAR files are also zip files, so you can extract the DTD from the JAR using Java's "jar" utility or with any program that understands zip format.)
The easiest way to learn what setups look like in XML, though, is to author a few of them in BehaviorSpace's GUI, save the model, and then examine the resulting .nlogo file in a text editor. The experiment setups are stored towards the end of the .nlogo file, in a section that begins and ends with a experiments tag. Example:
<experiments>
<experiment name="experiment" repetitions="10" runMetricsEveryTick="true">
<setup>setup</setup>
<go>go</go>
<exitCondition>not any? fires</exitCondition>
<metric>burned-trees</metric>
<enumeratedValueSet variable="density">
<value value="40"/>
<value value="0.1"/>
<value value="70"/>
</enumeratedValueSet>
</experiment>
</experiments>
In this example, only one experiment setup is given, but you can put as many as you want between the beginning and ending experiments tags.
Between looking at the DTD, and looking at examples you create in the GUI, it will hopefully be apparent how to use the tags to specify different kind of experiments. The DTD specifies which tags are required and which are optional, which may be repeated and which may not, and so forth.
When XML for experiment setups is included in a model file, it does not begin with any XML headers, because not the whole file is XML, only part of it. If you keep experiment setups in their own file, separate from the model file, then the extension on the file should be .xml not .nlogo, and you'll need to begin the file with proper XML headers, as follows:
<?xml version="1.0" encoding="us-ascii"?> <!DOCTYPE experiments SYSTEM "behaviorspace.dtd">
The second line must be included exactly as shown. In the first line, you may specify a different encoding than us-ascii, such as UTF-8, but NetLogo doesn't support non-ASCII characters in most situations, so specifying a different encoding may be pointless.
If BehaviorSpace is not sufficient for your needs, a possible alternative is to use our Controlling API, which lets you write Java code that controls NetLogo. The API lets you run BehaviorSpace experiments from Java code, or, you can write custom code that controls NetLogo more directly to do BehaviorSpace-like things. See the Controlling section of the User Manual for further details on both possibilities.
BehaviorSpace is still under development. We'd like to hear from you about what what additional features would be useful to you in your work. Please write us at feedback@ccl.northwestern.edu.