Releases | Open Source Science

https://github.com/activitysim/activitysim - v1.4.0

This release incorporates several minor changes to the ActivitySim platform.

Notable Changes

The following are changes that may alter results for existing models (generally for the better)...

SANDAG abm3 v15.0.0 contributions by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/934. This fixes a zone sampling bug in the disaggregate accessibility model.
Update to use pandas v2.* by @jpn-- in https://github.com/ActivitySim/activitysim/pull/932. This makes ActivitySim compatible with pandas v2. The migration from pandas 1.x to 2.x is mostly a seamless and transparent transition, where a handful of method names have changed under the hood but the functionality of the system is not impacted. However, a few changes, in particular the indexing and sort order of merged DataFrames, resulted in some changes in the reproducible random numbers, which required regenerating regression tests. Users updating existing models may see different results due to differences in the random numbers, but results should still be statistically valid, and will be stable and predictable after the change in the same way they were before the change.

Other Changes

There are a number of other updates that should not impact existing models in any way...

refactor(shadow_pricing.py): remove duplicate default_segment_to_name_dict by @asiripanich in https://github.com/ActivitySim/activitysim/pull/930
Restore input checker docs by @jpn-- in https://github.com/ActivitySim/activitysim/pull/941
Adding do_skim to tour mode choice by @dhensle in https://github.com/ActivitySim/activitysim/pull/943
Pin larch by @jpn-- in https://github.com/ActivitySim/activitysim/pull/947
Minor typo/bug fixes by @dkyleward in https://github.com/ActivitySim/activitysim/pull/945

New Contributors

@dkyleward made their first contribution in https://github.com/ActivitySim/activitysim/pull/945

Full Changelog: https://github.com/ActivitySim/activitysim/compare/v1.3.4...v1.4.0

- Jupyter Notebook
Published by jpn-- 9 months ago

https://github.com/activitysim/activitysim - v1.3.4

What's Changed

GitHub Actions updates by @jpn-- in https://github.com/ActivitySim/activitysim/pull/926
Tracing fix for HH ID that exists in both synthetic and proto populations by @dhensle in https://github.com/ActivitySim/activitysim/pull/901
release instructions by @jpn-- in https://github.com/ActivitySim/activitysim/pull/927
Use pandas 2 for docbuild environment by @jpn-- in https://github.com/ActivitySim/activitysim/pull/928

Full Changelog: https://github.com/ActivitySim/activitysim/compare/v1.3.2...v1.3.4

- Jupyter Notebook
Published by jpn-- 12 months ago

https://github.com/activitysim/activitysim - v1.3.2

This bug fix release should fix problems with changing the location of the sharrow cache directory.

What's Changed

Fix installer by @jpn-- in https://github.com/ActivitySim/activitysim/pull/887
Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @dependabot in https://github.com/ActivitySim/activitysim/pull/888
Sharrow Cache Dir Setting by @dhensle in https://github.com/ActivitySim/activitysim/pull/893

New Contributors

@dependabot made their first contribution in https://github.com/ActivitySim/activitysim/pull/888

Full Changelog: https://github.com/ActivitySim/activitysim/compare/v1.3.1...v1.3.2

- Jupyter Notebook
Published by jpn-- over 1 year ago

https://github.com/activitysim/activitysim - v1.3.1

Version 1.3.1 fixes the list of installation dependencies, and is otherwise identical to 1.3.0.

ActivitySim version 1.3 brings some significant new features to the platform. Users may need to make some small changes to configuration files to take full advantage of the version.

New Canonical Examples

Beginning with version 1.3, ActivitySim provides two supported "canonical" example implementations:

the SANDAG Model is a two-zone model based on the SANDAG ABM3 model, and
the MTC Model is a one-zone model based on the MTC's Travel Model One.

Each example implementation includes a complete set of model components, input data, and configuration files, and is intended to serve as a reference for users to build their own models. They are provided as stand-alone repositories, to highlight the fact that model implementations are separate from the ActivitySim core codebase, and to make it easier for users to fork and modify the examples for their own use without needing to modify the ActivitySim core codebase. The examples are maintained by the ActivitySim Consortium and are kept up-to-date with the latest version of ActivitySim.

{note} The two example models are not identical to the original agency models from which they were created. They are generally similar to those models, and have been calibrated and validated to reproduce reasonable results. They are intended to demonstrate the capabilities of ActivitySim and to provide a starting point for users to build their own models. However, they are not intended to be used as-is for policy analysis or forecasting.

Logging

The reading of YAML configuration files has been modified to use the "safe" reader, which prohibits the use of arbitrary Python code in configuration files. This is a security enhancement, but it requires some changes to the way logging is configured.

In previous versions, the logging configuration file could contain Python code to place log files in various subdirectories of the output directory, which might vary for different subprocesses of the model, like this:

yaml logging: handlers: logfile: class: logging.FileHandler filename: !!python/object/apply:activitysim.core.config.log_file_path ['activitysim.log'] mode: w formatter: fileFormatter level: NOTSET

In the new version, the use of !!python/object/apply is prohibited. Instead of using this directive, the log_file_path function can be invoked in the configuration file by using the get_log_file_path key, like this:

yaml logging: handlers: logfile: class: logging.FileHandler filename: get_log_file_path: activitysim.log mode: w formatter: fileFormatter level: NOTSET

Similarly, previous use of the if_sub_task directive in the logging level configuration like this:

yaml logging: handlers: console: class: logging.StreamHandler stream: ext://sys.stdout level: !!python/object/apply:activitysim.core.mp_tasks.if_sub_task [WARNING, NOTSET] formatter: elapsedFormatter

can be replaced with the if_sub_task and if_not_sub_task keys, like this:

yaml logging: handlers: console: class: logging.StreamHandler stream: ext://sys.stdout level: if_sub_task: WARNING if_not_sub_task: NOTSET formatter: elapsedFormatter

For more details, see logging.

Chunking

Version 1.3 introduces a new "explicit" chunking mechanism.

Explicit chunking is simpler to use and understand than dynamic chunking, and in practice has been found to be more robust and reliable. It requires no "training" and is activated in the top level model configuration file (typically settings.yaml):

yaml chunk_training_mode: explicit

Then, for model components that may stress the memory limits of the machine, the user can specify the number of choosers in each chunk explicitly, either as an integer number of choosers per chunk, or as a fraction of the overall number of choosers. This is done by setting the explicit_chunk configuration setting in the model component's settings. For this setting, integer values greater than or equal to 1 correspond to the number of chooser rows in each explicit chunk. Fractional values less than 1 correspond to the fraction of the total number of choosers. If the explicit_chunk value is 0 or missing, then no chunking is applied for that component. The explicit_chunk values in each component's settings are ignored if the chunk_training_mode is not set to explicit. Refer to each model component's configuration documentation for details.

Refer to code updates that implement explicit chunking for accessibility in PR #759, for vehicle type choice, non-mandatory tour frequency, school escorting, and joint tour frequency in PR #804, and all remaining interaction-simulate components in PR #870.

Automatic dropping of unused columns

Variables that are not used in a model component are now automatically dropped from the chooser table before the component is run. Whether a variable is deemed as "used" is determined by a text search of the model component code and specification files for the variable name. Dropping unused columns can be disabled by setting drop_unused_columns to False in the compute_settings for any model component, but by default this setting is True, as it can result in a significant reduction in memory usage for large models.

Dropping columns may also cause problems if the model is not correctly configured. If it is desired to use this feature, but some required columns are being dropped incorrectly, the user can specify columns that should not be dropped by setting the protect_columns setting under compute_settings. This allows the user to specify columns that should not be dropped, even if they are not apparently used in the model component. For example:

yaml compute_settings: protect_columns: - origin_destination

Code updates to drop unused columns are in PR #833 and to protect columns in PR #871.

Automatic conversion of string data to categorical

Version 1.3 introduces a new feature that automatically converts string data to categorical data. This reduces memory usage and speeds up processing for large models. The conversion is done automatically for string columns in most chooser tables.

To further reduce memory usage, there is also an optional downcasting of numeric data available. For example, this allows storing integers that never exceed 255 as int8 instead of int64. This feature is controlled by the downcast_int and downcast_float settings in the top level model configuration file (typically settings.yaml). The default value for these settings is False, meaning that downcasting is not applied. It is recommended to leave these settings at their default values unless memory availability is severely constrained, as downcasting can cause numerical instability in some cases. First, changing the precision of numeric data could cause results to change slightly and impact a previous calibrated model result. Second, downcasting to lower byte data types, e.g., int8, can cause numeric overflow in downstream components if the numeric variable is used in mathematical calculations that would result in values beyond the lower bit width limit (e.g. squaring the value). If downcasting is desired, it is strongly recommended to review all model specifications for compatability, and to review model results to verify if the changes are acceptable.

See code updates in PR #782 and PR #863

Alternatives preprocessors for trip destination.

Added alternatives preprocessor in PR #865, and converted to separate preprocessors for sample (at the TAZ level) and simulate (at the MAZ level for 2 zone systems) in PR #869.

Per-component sharrow controls

This version adds a uniform interface for controlling sharrow optimizations at the component level. This allows users to disable sharrow entirely, or to disable the "fastmath" optimization for individual components. Controls for sharrow are set in each component's settings under compute_settings. For example, to disable sharrow entirely for a component, use:

yaml compute_settings: sharrow_skip: true

This overrides the global sharrow setting, and is useful if you want to skip sharrow for particular components, either because their specifications are not compatible with sharrow or if the sharrow performance is known to be poor on this component.

When a component has multiple subcomponents, the sharrow_skip setting can be a dictionary that maps the names of the subcomponents to boolean values. For example, in the school escorting component, to skip sharrow for an OUTBOUND and OUTBOUND_COND subcomponent but not the INBOUND subcomponent, use the following settings:

yaml compute_settings: sharrow_skip: OUTBOUND: true INBOUND: false OUTBOUND_COND: true

The compute_settings can also be used to disable the "fastmath" optimization. This is useful if the component is known to have numerical stability issues with the fastmath optimization enabled, usually when the component potentially works with data that includes NaN or Inf values. To disable fastmath for a component, use:

yaml compute_settings: fastmath: false

Code updates that apply these settings are in PR #824.

Configuration validation

Version 1.3 adds a configuration validation system using the Pydantic library. Previously, the YAML-based configuration files were allowed to contain arbitrary keys and values, which could lead to errors if the configuration was not correctly specified. The new validation system checks the configuration files for correctness, and provides useful error messages if the configuration is invalid. Invalid conditions include missing required keys, incorrect data types, and the presence of unexpected keys. Existing models may need to be cleaned up (i.e. extraneous settings in config files removed) to conform to the new validation system.

See PR #758 for code updates.

Input checker

Version 1.3 adds an input checker that verifies that the input data is consistent with expectations. This tool can help identify problems with the input data before the model is run, and can be used to ensure that the input data is correctly formatted and complete.

See PR #753 for code updates.

Removal of orca dependency

This new version of ActivitySim does not use orca as a dependency, and thus does not rely on orca’s global state to manage data. Instead, a new State class is introduced, which encapsulates the current state of a simulation including all data tables. This is a significant change “under the hood”, which may be particularly consequential for model that use “extensions” to the ActivitySim framework. See PR #654 for code updates.

Pull Requests included in this release

Black 2022 style by @jpn-- in https://github.com/ActivitySim/activitysim/pull/642
add modelsettings to estimator.writecoefficients by @bwentl in https://github.com/ActivitySim/activitysim/pull/651
BayDAG Contributions by @dhensle in https://github.com/ActivitySim/activitysim/pull/657
docs: apply a minor correction to user guides by @asiripanich in https://github.com/ActivitySim/activitysim/pull/659
Trip scheduling logic by @jpn-- in https://github.com/ActivitySim/activitysim/pull/660
Pin Dependencies by @jpn-- in https://github.com/ActivitySim/activitysim/pull/665
Updated SEMCOG Example by @dhensle in https://github.com/ActivitySim/activitysim/pull/603
syncronize by @jpn-- in https://github.com/ActivitySim/activitysim/pull/680
Replace ORCA with non-global state by @jpn-- in https://github.com/ActivitySim/activitysim/pull/654
Fix memory usage by @jpn-- in https://github.com/ActivitySim/activitysim/pull/751
orca residual cleanup by @jpn-- in https://github.com/ActivitySim/activitysim/pull/694
Disable unstable estimation mode test by @jpn-- in https://github.com/ActivitySim/activitysim/pull/765
Overflow protection by @jpn-- in https://github.com/ActivitySim/activitysim/pull/764
Fixes windows error on large MAZ systems by @jpn-- in https://github.com/ActivitySim/activitysim/pull/760
update repo test pointers by @jpn-- in https://github.com/ActivitySim/activitysim/pull/783
Input Checker by @dhensle in https://github.com/ActivitySim/activitysim/pull/753
Merge missing changes from main back to develop by @jpn-- in https://github.com/ActivitySim/activitysim/pull/788
Config Settings and Documentation by @jpn-- in https://github.com/ActivitySim/activitysim/pull/758
Explicit chunking by @jpn-- in https://github.com/ActivitySim/activitysim/pull/759
Selecting choices from joint tour participant ID column explicitly; by @bricegnichols in https://github.com/ActivitySim/activitysim/pull/653
Option to write output tables as parquet files by @stefancoe in https://github.com/ActivitySim/activitysim/pull/763
rollback mistaken merge by @jpn-- in https://github.com/ActivitySim/activitysim/pull/793
Moved 'tot_tours' from nm tour frequency script to alternatives file by @JoeJimFlood in https://github.com/ActivitySim/activitysim/pull/661
Add options to handle larger dataset for location models by @bwentl in https://github.com/ActivitySim/activitysim/pull/687
Pydantic 2 by @jpn-- in https://github.com/ActivitySim/activitysim/pull/796
added stricter joining of annotated fields by @nick-fournier-rsg in https://github.com/ActivitySim/activitysim/pull/672
Vehicle Type Alts Filtering Bug Fix by @dhensle in https://github.com/ActivitySim/activitysim/pull/790
Data Type Optimization by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/782
Alt col name bug fix for option to handle larger dataset for location models by @bwentl in https://github.com/ActivitySim/activitysim/pull/798
Performance monitoring fixes by @jpn-- in https://github.com/ActivitySim/activitysim/pull/797
(pre) release for 1.3 by @jpn-- in https://github.com/ActivitySim/activitysim/pull/800
cleanup_failed_trips requires workflow state by @jpn-- in https://github.com/ActivitySim/activitysim/pull/801
Vehicle Type Optimization by @dhensle in https://github.com/ActivitySim/activitysim/pull/806
remove code duplication by @jpn-- in https://github.com/ActivitySim/activitysim/pull/803
Fix at-work frequency by @jpn-- in https://github.com/ActivitySim/activitysim/pull/808
generate stable vehicle type dtype categories by @jpn-- in https://github.com/ActivitySim/activitysim/pull/807
explicit chunking for interaction-simulate components by @jpn-- in https://github.com/ActivitySim/activitysim/pull/804
use default versioning scheme, not simplified by @jpn-- in https://github.com/ActivitySim/activitysim/pull/809
No categories for escort participants by @jpn-- in https://github.com/ActivitySim/activitysim/pull/813
School Escorting Optimization by @dhensle in https://github.com/ActivitySim/activitysim/pull/810
Fix inconsistent alts object name by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/817
Time period to categorical in legacy mode by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/819
Infrastructure Updates by @jpn-- in https://github.com/ActivitySim/activitysim/pull/812
Automatically ignore output directories from workflow runs by @jpn-- in https://github.com/ActivitySim/activitysim/pull/829
Vehicle type Categorical by @jpn-- in https://github.com/ActivitySim/activitysim/pull/826
BayDAG Contribution #1: Auto Ownership Processing by @dhensle in https://github.com/ActivitySim/activitysim/pull/767
BayDAG Contribution #6: Joint Tour Participation Infinite Loop by @dhensle in https://github.com/ActivitySim/activitysim/pull/772
BayDAG Contribution #4: Joint Tour Destination Performance by @dhensle in https://github.com/ActivitySim/activitysim/pull/770
BayDAG Contribution #8: Landuse and Reindex available in location choice by @dhensle in https://github.com/ActivitySim/activitysim/pull/774
BayDAG Contribution #9: Mode Choice Logsum Extraction by @dhensle in https://github.com/ActivitySim/activitysim/pull/775
BayDAG Contribution #2: Enhanced Disaggregate Accessibility Merging by @dhensle in https://github.com/ActivitySim/activitysim/pull/768
BayDAG Contribution #15: Larch Interface for Added Models by @dhensle in https://github.com/ActivitySim/activitysim/pull/781
BayDAG Contribution #5: Joint Tour Frequency and Composition Estimation by @dhensle in https://github.com/ActivitySim/activitysim/pull/771
BayDAG Contribution #7: Sampling in EDB for Location Choice by @dhensle in https://github.com/ActivitySim/activitysim/pull/773
Changed Environment YML by @lachlan-git in https://github.com/ActivitySim/activitysim/pull/832
BayDAG Contribution #3: CDAP estimation with joint component by @dhensle in https://github.com/ActivitySim/activitysim/pull/769
BayDAG Contribution #10: NMTF Person Available Periods by @dhensle in https://github.com/ActivitySim/activitysim/pull/776
BayDAG Contribution #13: Estimation Mode Usability by @dhensle in https://github.com/ActivitySim/activitysim/pull/779
BayDAG Contribution #14: Increasing Larch Loading for NMTF by @dhensle in https://github.com/ActivitySim/activitysim/pull/780
pin pandera by @jpn-- in https://github.com/ActivitySim/activitysim/pull/843
fix bug in interaction_simulate by @jpn-- in https://github.com/ActivitySim/activitysim/pull/839
BayDAG Contribution #11: School Escorting Estimation Updates by @dhensle in https://github.com/ActivitySim/activitysim/pull/777
BayDAG Contribution #16: Parking Locations in Trip Matrices by @dhensle in https://github.com/ActivitySim/activitysim/pull/840
Common settings for fastmath, sharrow_skip, and other compute controls across components by @jpn-- in https://github.com/ActivitySim/activitysim/pull/824
BayDAG All Contributions by @dhensle in https://github.com/ActivitySim/activitysim/pull/834
Automatically drop unneeded columns in choosers table by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/833
add nd_skims for sharrow flows on trip dest by @jpn-- in https://github.com/ActivitySim/activitysim/pull/852
Add tolerance control on progressive testing by @jpn-- in https://github.com/ActivitySim/activitysim/pull/855
allow input checker aux files to be parquet by @jpn-- in https://github.com/ActivitySim/activitysim/pull/851
Various minor fixes by @jpn-- in https://github.com/ActivitySim/activitysim/pull/856
School Escorting Runtime Optimization by @dhensle in https://github.com/ActivitySim/activitysim/pull/828
Allow skim access in trip mode choice annotate by @dhensle in https://github.com/ActivitySim/activitysim/pull/857
Consecutive time window overlap bug fix by @dhensle in https://github.com/ActivitySim/activitysim/pull/854
state.run.all should allow memory profiling by @jpn-- in https://github.com/ActivitySim/activitysim/pull/859
ensure tour_type is categorical by @jpn-- in https://github.com/ActivitySim/activitysim/pull/863
Trip Destination Alternatives Preprocessor by @dhensle in https://github.com/ActivitySim/activitysim/pull/865
Trip destination alts preprocessor for both sample and simulate steps by @dhensle in https://github.com/ActivitySim/activitysim/pull/869
protect_columns for all simulations (choosers, alts, simple simulate, interaction simulate, etc) by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/871
joint_tour_frequency_composition column to categorical by @i-am-sijia in https://github.com/ActivitySim/activitysim/pull/866
Explicit chunking on all interaction simulate models by @dhensle in https://github.com/ActivitySim/activitysim/pull/870
require numpy<2.x by @jpn-- in https://github.com/ActivitySim/activitysim/pull/875
Changes to support sharrow on 2-zone model by @jpn-- in https://github.com/ActivitySim/activitysim/pull/867
Integer Encoding Vehicle Allocation Spec by @dhensle in https://github.com/ActivitySim/activitysim/pull/877
Performance docs by @jpn-- in https://github.com/ActivitySim/activitysim/pull/878
Release notes v1.3 by @jpn-- in https://github.com/ActivitySim/activitysim/pull/882
Bugfix for trip scheduling choice by @jpn-- in https://github.com/ActivitySim/activitysim/pull/884
Fix docs, add performance report by @jpn-- in https://github.com/ActivitySim/activitysim/pull/885

New Contributors

@bwentl made their first contribution in https://github.com/ActivitySim/activitysim/pull/651
@asiripanich made their first contribution in https://github.com/ActivitySim/activitysim/pull/659
@bricegnichols made their first contribution in https://github.com/ActivitySim/activitysim/pull/653
@stefancoe made their first contribution in https://github.com/ActivitySim/activitysim/pull/763
@lachlan-git made their first contribution in https://github.com/ActivitySim/activitysim/pull/832

Full Changelog: https://github.com/ActivitySim/activitysim/compare/v1.2.2...v1.3.0

- Jupyter Notebook
Published by jpn-- over 1 year ago

https://github.com/activitysim/activitysim - v1.3.0

ActivitySim version 1.3 brings some significant new features to the platform. Users may need to make some small changes to configuration files to take full advantage of the version.

New Canonical Examples

Beginning with version 1.3, ActivitySim provides two supported "canonical" example implementations:

the SANDAG Model is a two-zone model based on the SANDAG ABM3 model, and
the MTC Model is a one-zone model based on the MTC's Travel Model One.