bioflow

Information Flow Analysis toolkit for biological networks

https://github.com/chiffa/bioflow

Repository

Information Flow Analysis toolkit for biological networks

README.rst

|PyPI license| |PyPI Python version| |PyPI version| |PyPI status|

BioFlow Project
===============

Information Flow Analysis in biological networks

|Build Status| |Docs| |Coverage Status|

Overview:
---------

BioFlow is a quantitative systems biology tool.

It leverages information complexity theory in order to generate mechanism-of-action
hypotheses in a robust, quantified, highly granular, explainable, customizeable and near-optimal
manner.

The core of the BioFlow is a biological knowledge entity relationship graph. It is  derived from
existing biological knowledge repositories, such as Reactome, Gene Ontology, HINT, Phosphosite,
ComplexPortal or BioGrid. Upon importing them, BioFlow converts relationship

The biological knowledge entities from different repositories are cross-referenced and
connections between them established based on how likely they are to be directly connected in an
hypothesis for a general mechanism of action.

The input for BioFlow is a list of physical entities most likely involved in a process of
interest - usually a list of genes or proteins.

BioFlow then examines all the possible paths linking those physical entities of interest and
assigns an overall probability to all the paths, then proceeding to analyse the most likely
common denominators for paths and assigning to each biological knowledge entity and relationship
between them a chance of being part of the mechanism-of-action behind the process.

In order to avoid picking up on spurious associations, the resulting weights for biological
knowledge entity and relationships between them are compared to the ones generated by random
groups of genes and only based on that the final probabilities of inclusion into hypotheses for
the mechanism-of-action are calculated.

The final probabilities for the knowledge entities are sorted by the probability of inclusion and
are printed to the terminal and saved as a .tsv file, whereas the entire network with probabilities
assigned to entities and relationships between them is saved for further analysis as a .gdf file.

BioFlow is:

- **robust**,
  because we are weighting the relationships between the knowledge entities based
  on how likely they are to be real and be included in a mechanism of action. Thanks to that, we
  can include relationships we are not sure of (eg Y2H protein-protein interactions with low
  confidence score) and by assigning them a low weight make sure they will not be included into
  mechanisms of action hypothesis unless no other possible connection can explain a phenotype.

- **quantified**,
  because every single biological knowledge entity is assigned both a weight
  and a p-value for how likely it is to contribute to a mechanism of action.

- **highly granular
  because thanks to the weighting of edges and inclusion of multiple
  sources of biological knowledge, we can evaluate in the same pass steps for which we have a very
  good mecanistical understanding (eg specific cite phosphorilation on a specific isoform by
  another specific isoform) as well as steps for which we are not entirely convinced of in-vivo
  existence (once again, low score Y2H). Similarly, thanks to a unified model of biological entity

- **explainable**,
  because all the steps in the hypothesis are biological entities and
  connections between them. As such, a direct translation from a BioFlow analysis to an experiment
  is "let's suppress this entity/relationship with a high p-value and a high weight" and see if it
  affects our process.

- **customizable**,
  because BioFlow is build as a library with multiple abstraction levels and customization
  capabilities.

  - The knowledge graph can be directly accessed and modified through
    the graphical user interface provided by the `neo4j graph database `__
    storage back-end, as well as extracted as ``scipy.sparse`` weighted graph laplacian and
    adjacency matrices with index-to-entity-id maps.
  - By adding new rdf turtle tuple parsers into the ``bioflow.bio_db_parser`` and inserters into
    the ``bioflow.db_importers``, new sources of biological knowledge can be integrated.
  - By modifying routines in the ``bioflow.algorithms_bank``, new entity
    relationship weighting modes, background sampling algorithms or evaluation methods of hypotheses
    statistical significance can be introduced.
  - In case of absolute need, alternative storage backends can be implemented by
    re-implementing the ``GraphDBPipe`` object in ``bioflow.neo4j_db.cypher_drivers`` or methods
    from ``bioflow.sample_storage.mongodb``.

- **near-optimal**,
  because we are using a finite-world version of Solomonoff Algorithmic
  Inference - a provably optimal way of learning model representation from data. In order to
  accelerate  the computation and stabilize the output with respect to the noise often encountered
  in biological data, we slightly modify the inference mode and background computation algorithm,
  hence the "near"-optimality.


Examples of applications:
-------------------------

A good way of thinking about BioFlow is as a specialized search engine for biological knowledge

Molecular mechanism hypotheses from high-throughput experiment results:
```````````````````````````````````````````````````````````````````````

A lot of modern biology exploring phenotypes of interest from an unbiased perspective. In
order to get an initial idea of a mechanism, a standard approach is to retrieve several
experimental models that present a deviation of a phenotype from the model and perform a
high-throughput experiment, such as mutagenisis, mutant library screen, transcription profiling
or proteomics profiling. A result is often a list of hundreds of genes that are potentially
involved in a mechanism, but it is rare that a mechanism is directly visible or even if there is
any certainty that the list is not mostly composed of experimental artefacts.

BioFlow is capable of generating an unbiased, quantified list of hypotheses as to the
molecular mechanism underlying the process.

- Thanks to its integrated knowledge graph, it is
  capable to implicate mechanisms that would not be detectable by the screening method used to
  generate input data.
- Thanks to its parallel evaluation of possible molecular mechanisms, it can point to backup
  mechanisms as well as molecular entities or pathways weekly involved in the process of interest.
- Thanks to its null model of a random list of genes, it is capable to filter out random
  nodes that are due to artifacts.
- Thanks to its null model, if the provided list of genes is a pure artefact, it will not
  call any nodes as likely paths and mark all hypotheses of molecular mechanisms as insufficiently
  likely.


Personalized cancer medicine:
`````````````````````````````
Cancers are characterized by a large variety of mutation affecting numerous pathways in the
bioilogical organisms. Some of the effects are antagonistic, some synergetic, but a lot of
mutations/expression modifications are variations affecting similar pathways.

BioFlow is capable of integrating the list of perturbations found in the patients cancer (such as
mutations, transcriptional modification, protein trafficking imbalance, ...) and build a model of
perturbed molecular pathways in the patient, allowing the drugs and drug combination selection to
be prioritized.


Evaluation of the effects of large-scale genome perturbation:
`````````````````````````````````````````````````````````````
In some cases, such as partial genome duplication or aneuploidy, or a recombination event, a large
number of gene expression levels or protein structures are perturbed simultaneously. While
models might exist for single genes or small groups of genes, the sheer number of perturbations
and ways in which they can interfere makes the prediction intractable for humans.

In case any group of perturbation is likely to have a major synergistic effect, BioFlow will
highlight them as well as the likely molecular mechanism they could act through.


In-silico drug secondary effects prediction:
````````````````````````````````````````````
Given that a lot of small-molecule drugs possess a polypharmacological multi-target activity, a
number of secondary effects that cannot be traced to single targets being poisoned by the
compound or its metabolic derivatives, are hypothesized to be related to the systemic effects of
unspecific binding of the compound.

By combining the list of compounds that have been associated to a specific secondary effect as
well as the binding profiles for those compounds (either measured in-vitro or simulated
in-silico), BioFlow can create the network of the nodes and relationships most likely implicated in
a mechanism of action underlying the secondary effect induction by off-target binding. By
comparing the network flow generated by a de-novo compound binding profile to the ones engendered
by the drugs presenting the secondary effect or not, we can infer most likely secondary effects
to prioritize donwstream pre-clinical testing.


In-silico drug repurposing:
```````````````````````````
One of the advantages of the readily approved drugs is that they have been shown to be relatively
safe in humans and have well-understood secondary effects. As such, their application in
treatments of novel diseases is significantly more desireable than development of new compounds,
both due a shorter time of development and only efficacy trails in humans being needed. This
application is particularly interesting for rare and neglected diseases, where a de-novo compound
development is usually not economically viable.

BioFlow can be used to construct the profile of biological entities that are most likely to be
implicated in the molecular mechanism of action behind the disease (such as deviation from nor in
human rare genetic disorders or essential pathways in pathogens). Based on the in-silico or in-vivo
binding assays of approved compounds to the targets relevant to the phenotype, BioFlow can help
to prioritize the compounds for further investigation.



Relationship to other methods:
------------------------------

Network diffusive models:
`````````````````````````
BioFlow generalizes the network diffusion models. While both BioFlow and network diffusion models
rely on the graph laplacian and the flow through it, BioFlow uses that formalization to rank the
most possible molecular mechanisms in a maximally unbiased, nearly optimal manner, whereas most
network diffusion models work by pure analogy.

BioFlow's near-optimality provides an explanation to the uncanny efficiency of the graph
diffusion models, but in addition provides a direct interpretability of the results as well as
suggests schemes for weighting the graph edges.


Network topology methods:
`````````````````````````
Compared to the network topology methods, thanks to its weighting scheme and all-paths
probability evaluation, BioFlow is much less brittle to the inclusion of low-confidence edges
affecting biological topology. It allows for multiple abstraction levels to be examined
simultaneously, which is particularly interested in cases where a granular information is
available only for a small subset of nodes and edges. In turn, this capability to work with
multiple levels of abstraction granularity allows BioFlow to work with heterogeneous data,
integrating different types of perturbation at the same time.


Annotation group methods:
`````````````````````````
Given that BioFlow does not rely on strict borders between categories, uses weights and
simultaneously evaluates all the possible molecular mechanisms based on the data, it is
significantly less brittle with regards to specific molecular entity inclusion in the annotation
groups or the inclusion of specific molecular entities into the list associated to the process of
interest.

Similarly, when BioFlow analyses the possible hypotheses for human-generated annotation networks,
it provides much more interepretability with regards to the connection to the annotation
proximity of terms, taking in account annotation terms overloading, single molecular entity
over-annotation or confidence of molecular entity annotation with an annotation term.

Finally, by combining multiple annotation networks analysis with the molecular entity network, it
is less prone to neglecting processes that have not yet been annotated in the annotation network.

Mechanistic models:
```````````````````
Given that BioFlow is capable to operate with multiple levels of granularity of biological
knowledge abstractions and uses a unified model for all the molecular entities and relationships
between them, BioFlow is able to work with many more data types, does not require exact knowledge
to generate hypotheses and is computationally more simple and stable at scale. Similarly, it is
able to suggest possible mechanisms where there are yet no mechanistic models.

However, BioFlow's model means that it has a more restrictetd expressivity. For instance, it will
not be able to recognize synergistic vs antagonistic interaction between perturbations it is
analyzing or distinguish repressors from inducers.

Overall, BioFlow is a good precursor to mechanistic models, if nodes that and interactions that
are ranked highly by BioFlow have a strong overlap with known mechanistic models.


Functioning and specifics of the implementation:
------------------------------------------------

BioFlow requires an instance of `neo4j graph database `__ running for the
main knowledge repository, as well as an instance of the `MongoDB `__.

Upon start, BioFlow will look for ``$BIOFLOWHOME`` environment variable to know where to store its
files. If none found, it will use the default ``~/bioflow`` directory.

Inside the ``$BIOFLOWHOME`` it will store the user configs .yaml file
(``$BIOFLOWHOME/configs/main_configs.yaml``). If for whatever reason it doesn't find them, it will
copy the default configs it has there. If you want to reset configs to default, just delete or
rename your ``configs.yaml`` file.

The config contains several sections:

- ``DB_locations``: maps where to look for the databases it uses to build the main biological
  entity relationship graph and where to store them locally. If you get an error on download,
  chances are one of the source databases has moved. Alternatively, if you want to use a specific
  snapshot of the database, you can change the online location the file is loaded from.
- ``Servers``: stores the urls and ports BioFlow will expect MongoDB and Neo4j to be available.
- ``Sources``: allows to select the organism. If you are not sure of what you are doing, just
  uncomment the organism you want to work on.
- ``User_settings``:

  - ``smtp_logging``: enable and configure if you want to receive notifications about errors or
    run finishing by mail. Given you will need a local smtp server sending mails properly, setting
    this section is for advanced users only.
  - ``environement``: modifies how some aspects of BioFlow work. Comments in ``configs.yaml``
    explain what they do. Most of them can be overriden in direct calls
  - ``analysis``: controls the parameters used to calculate statistical significance
  - ``debug_flags``: potentially useful if you want to debug an issue or fill out a bug report.

Everything is logged to ``$BIOFLOWHOME/.internal/logs``. As such, debugs, critical errors and
warnings are all stored there.

Upon execution a run output folder in ``$BIOFLOWHOME/outputs/`` is created with the datetime ISO
name that will contain any output generated by the run, as well as info-level log (basically, a
copy of what is printed on the console).

Finally, due to large differences in topological structure and weighting algorithms, the analysis of
biological knowledge nodes that represent molecular entities (proteins, isoforms, small molecules)
and the ones that represent human-made abstractions to reason about them (Gene Ontology terms,
Pathways, ...) are split into two different modules (molecular network/Interactome vs annotation
network/BioKnowledge modules/classes).

The full API documentation and more in-depth guides are available at `readthedocs.org
`__.


Basic Usage:
============

Installation walk-through:
--------------------------

Ubuntu direct installation:
```````````````````````````

1) Install the Anaconda python 3.X and use the python provided by Anaconda python in all that
follows. A way of doing it is by Making Anacoda Python your default Python. The full process is
explained
`here `__.

2) Install libsuitesparse::

    > apt-get -y install libsuitesparse-dev

3) Install neo4j::

    > wget -O - https://debian.neo4j.org/neotechnology.gpg.key | sudo apt-key add -
    > echo 'deb https://debian.neo4j.org/repo stable/' | sudo tee /etc/apt/sources.list.d/neo4j.list
    > sudo apt-get update
    > sudo apt-get install neo4j

4) Install MongDB (Assuming Linux 18.04 - if not, see `here `__)::

    > sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
    > echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
    > sudo apt-get update
    > sudo apt-get install -y mongodb-org

For more information, refer to the `installation guide
`__

5) Finally, install BioFlow through pip::

    > pip install BioFlow

Or, if you want to install it directly::

   > git clone https://github.com/chiffa/BioFlow.git
   > cd 
   > pip install -r requirements.txt

Docker:
```````

If you want to build locally (notice you need to issue docker commands with docker-enabled
users)::

   > git clone https://github.com/chiffa/BioFlow.git
   > cd 
   > docker build -t "bioflow" .
   > docker run bioflow
   > docker-compose build
   > docker-compose up -d


If you want to pull from dockerhub or don't have access to BioFlow installation directory::

   > wget https://github.com/chiffa/BioFlow/blob/master/docker-compose.yml
   > mkdir -p $BIOFLOWHOME/input
   > mkdir -p $BIOFLOWHOME/source
   > mkdir -p $BIOFLOWHOME/.internal/docker-mongo/db-data
   > mkdir -p $BIOFLOWHOME/.internal/docker-neo4j/db-data
   > docker-compose build
   > docker-compose up -d

Finally attach to the running container::

   > docker attach bioflow_bioflow_1

For working from docker, you will have to have ``$BIOFLOWHOME`` environment variable defined (by
default ``$HOME/bioflow``).

Scripts with which docker build was tested can be found in the ``docker_script.sh`` file.

For persistent storage, the data will be stored in the mapped volumes as follows:

.. list-table:: Volume mapping
   :widths: 25 25 50
   :header-rows: 1

   * - What
     - Docker
     - On disk
   * - neo4j data
     - /data (neo4j docker)
     - $BIOFLOWHOME/.internal/docker-neo4j/db-data
   * - mongodb data
     - /data/db (mongodb docker)
     - $BIOFLOWHOME/.internal/docker-mongo/db-data
   * - bioflow home
     - /bioflow
     - $BIOFLOWHOME
   * - inputs
     - /input
     - $BIOFLOWHOME/input


Usage walk-through:
-------------------

.. WARNING::
    While BioFlow provides an interface to download the databases programmatically, the databases are subject to Licenses and Terms that it's up to the end users to respect

For more information about data and config files, refer to the `data and database guide
`__

Environment variables and setup:
````````````````````````````````

In order to connect to the neo4j database, bioflow needs to have the neo4j authentication
password setup. For safety reasons, it is possible exclusively through the environment variable
``$NEOPASS``.

In addition to that, it is possible to setup where the bioflow will be store its data with the
``$BIOFLOWHOME``.



Python scripts:
```````````````
This is the recommended method for using BioFlow.

An example usage script is provided by ``bioflow.analysis_pipeline_example.py``.

First, let's pull the online databases::

   > from bioflow.utils.source_dbs_download import pull_online_dbs
   > pull_online_dbs()

Now, we can import the main knowledge repository database handlers and build the main database::

   > from bioflow.db_importers.import_main import destroy_db, build_db
   > build_db()

The building process will take a bit - up to a couple of hours.

Now, you can start using the BioFlow proper::

   > from bioflow.annotation_network.knowledge_access_analysis import auto_analyze as \
   >    knowledge_analysis
   > from bioflow.molecular_network.interactome_analysis import auto_analyze as interactome_analysis

   > hits_file = "/your/path/here.tsv"
   > background_file = "/your/other_path/here.tsv"

And get to  work: map the hits and the background genes to internal db IDs::

   > hit_ids, sec_ids, background_ids = map_and_save_gene_ids(hits_file, background_file)

BioFlow expects the tsv/csv for hits or background files to contain one hit per line, and will
attempt to map them to UNIPROT protein nodes (used as a backbone to cross-link imported
databases), based on the following identifier types:

 - Gene names
 - HGCN symbols
 - PDB Ids
 - ENSEMBL Ids
 - RefSeq IDs
 - Uniprot IDs
 - Uniprot accession numbers

(Re)build the laplacians (not required unless the knowledge structure in the main knowledge
database has changed)::

   > rebuild_the_laplacians()

Launch the analysis itself for the information flow in the interactome::

   > interactome_analysis(source_list=[hits_ids],
                          output_destinations=[``],     #optional
                          desired_depth=20,                                 #optional
                          processors=3,                                     #optional
                          background_list=background_bulbs_ids,             #optional
                          skip_sampling=False)                              #optional

Launch the analysis itself for the information flow in the annotation network (experimental)::

   > knowledge_analysis(source_list=[hits_ids],
                        output_destinations=[``],     #optional
                        desired_depth=20,                                 #optional
                        processors=3,                                     #optional
                        background_list=background_bulbs_ids,             #optional
                        skip_sampling=False)                              #optional

Where:

:hits_ids: list of hits
:output_destinations: names to provide for the output destination (by default numbered from 0)
:desired_depth: how many samples we would like to generate to compare against
:processors: how many threads we would like to launch in parallel (in general 3/4 works best)
:background_list: list of background Ids
:skip_sampling: if true, skips the sampling of background set and retrieves stored ones instead

BioFlow will print progress to the StdErr from then on and will output to the ``$BIOFLOWHOME``,
in a folder called ``outputs_YYYY-MM_DD ``:

- .gdf file with the flow network and relevance statistics (``Interactome_Analysis_output.gdf``)
- visualisation of information flow through nodes in the null vs hits sets based on the node degree
- list of strongest hits (``interactome_stats.tsv``) (printed to StdOut as well)

The .gdf file can be further analysed with more appropriate tools, such as for instance
`Gephi `__.

NOTE: there is also a possibility to provide a flow target set (secondary set) as well as to
assign weights to hits as well as the background. This information is available in `the usage guide
`__. You can as
well refer to the ``bioflow._integration_tests.py`` file for examples of how it would work with
provided data (glycogen biosynthesis in yeast).

Enabling the SMTP logging requires you to define appropriate smtp server variables in the configs
file and add the smtp handles to all the loggers you want to use it::

    > from bioflow.utils.log_behavior import get_logger
    > from bioflow.utils.smtp_log_behavior import mail_handler

    > log = get_logger(__name__)
    > log.addHandler(mail_handler)


Command line:
`````````````

Command line can either be invoked by python execution: ::

    > python -m bioflow.cli  [--options]

Or, in case of installation with pip, directly from a command line (assumed here): ::

    > bioflow  [--options]

Setup environment (likely to take a while to pull all the online databases): ::

    > bioflow downloaddbs
    > bioflow loadneo4j

In case if needed, perform a self-diagnosis of the database: ::

    > bioflow diagneo4j

Set the set of perturbed proteins on which we would want to base our analysis ::

    > bioflow mapsource /your/path/here.tsv --background=/your/other_path/here.tsv

Rebuild the laplacians ::

    > bioflow rebuildlaplacians


Perform the analysis::

    > bioflow analyze --matrix interactome --depth 24 --processors 3 --background True
                    --name=

    > bioflow analyze --matrix annotome --depth 24 --processors 3 --background True
                    --name=

Alternatively::

    > bioflow analyze --depth 24 --processors 3 --background True --name=

For all of them, if the smtp is configured, an ``--smtplog`` option will trigger the mail
start/stop/exception reporting

More information is available with::

    > bioflow --help

    > bioflow about

The results of analysis will be available in the output folder, and printed out to the ``std.out``.


Post-processing:
````````````````
The .gdf file format is one of the standard format for graph exchange. It contains the following
columns for the nodes:

- node ID
- information current passing through the node
- node type
- ``legacy_id``
- degree of the node
- whether it is present or not in the hits list (source; negative for secondary list)
- if the hits list was weighted, the weight; if a secondary list was present, it's weights, but
in negative
- ``p-value``, comparing the information flow through the node to the flow expected for the random
set of genes
- -log10(``p_value``) (``p_p-value``)
- ``rel_value`` (information flow relative to the flow expected for a random set of genes)
- ``std_diff`` (how many standard deviations above the flow for a random set of genes the flow from
a hits list is) (not a robust metrics)

The most common pipleine involves using `Gephi open graph visualization platform `__:

- Load the gdf file into gephy
- Filter out all the nodes with information flow below 0.05 (Filters > Atrributes > Range > current)
- Perform clustering (Statistics > Modularity > Randomize & use weights)
- Filter out all the nodes below a significance threshold (Filters > Attributes > Range > p-value)
- Set Color nodes based on the Modularity Class (Nodes > Colors > Partition > Modularity Class)
- Set node size based on p_p-value (Nodes > Size > Ranking > p_p-value )
- Set text color based on whether the node is in the hits list (Nodes > Text Color > Partition > source)
- Set text size based on p_p-value (Nodes > Text Size > Ranking > p_p-value)\
- Show the lables (T on the bottom left)
- Set labes to the legacy IDs (Notepad on the bottom)
- Perform a ForeAtlas Node Separation (Layout > Force Atlas 2 > Dissuade Hubs & Prevent Overlap)
- Adjust label size
- Adjust labels position (Layout > LabelAdjust)

More details on the specifics of the .gdf output file is available in the `advanced usage guide
`__.

For more details or usage as a library, refer to the `usage guide
`__.


.. |License Type| image:: https://img.shields.io/badge/license-BSD3-blue.svg
   :target: https://github.com/chiffa/BioFlow/blob/master/License-new_BSD.txt
.. |Build Status| image:: https://travis-ci.org/chiffa/BioFlow.svg?branch=master
   :target: https://travis-ci.org/chiffa/BioFlow
.. |Coverage Status| image:: https://coveralls.io/repos/chiffa/BioFlow/badge.svg?branch=master&service=github
   :target: https://coveralls.io/github/chiffa/BioFlow?branch=master


.. |Python version| image:: https://img.shields.io/badge/python-3.X-blue.svg
   :target: https://www.python.org/downloads/release/python-2715/
.. |Branch Status| image:: https://img.shields.io/badge/status-alpha-red.svg
   :target: https://www.python.org/downloads/release/python-2715/
.. |Docs| image:: https://readthedocs.org/projects/bioflow/badge/?version=latest
   :target: https://bioflow.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation Status

.. |PyPI version| image:: https://img.shields.io/pypi/v/BioFlow.svg
   :target: https://pypi.python.org/pypi/BioFlow/
.. |PyPI Python version| image:: https://img.shields.io/pypi/pyversions/BioFlow.svg
   :target: https://pypi.python.org/pypi/BioFlow/
.. |PyPI license| image:: https://img.shields.io/pypi/l/BioFlow.svg
   :target: https://pypi.python.org/pypi/BioFlow/
.. |PyPI status| image:: https://img.shields.io/pypi/status/BioFlow.svg
   :target: https://pypi.python.org/pypi/BioFlow/

Owner

GitHub Events

Committers

Last synced: over 2 years ago

Issues and Pull Requests

Last synced: 11 months ago

Packages