biosamples-v4

The source code for the new version of the EMBL-EBI BioSamples database

https://github.com/ebibiosamples/biosamples-v4

Repository

The source code for the new version of the EMBL-EBI BioSamples database

README.adoc

image:https://api.codacy.com/project/badge/Grade/d1d650940b1f4a6286607c1d06573090["Codacy code quality", link="https://www.codacy.com/app/danielvaughan/biosamples-v4?utm_source=github.com&utm_medium=referral&utm_content=EBIBioSamples/biosamples-v4&utm_campaign=Badge_Grade"]
image:https://quay.io/repository/ebibiosamples/biosamples-v4/status["Docker Repository on Quay", link="https://quay.io/repository/ebibiosamples/biosamples-v4"]

== BioSamples

BioSamples https://www.ebi.ac.uk/biosamples/ is an ELIXIR Core Deposition Database that stores and supplies descriptions and metadata about biological samples used in research and development by academia and industry. Samples are either 'reference' samples (e.g. from 1000 Genomes, HipSci, FAANG) or experimental samples, and have been used in an assay that has generated publicly available data in the European Nucleotide Archive (ENA) or ArrayExpress.

BioSamples supports links between sample records and any sample-derived datasets, including sequence-based datasets such as those held in ENA or ArrayExpress, -omics-based datasets such as those in PRIDE or MetaboLights, and any other assay-based databases. It provides links to assays and specific samples and accepts direct submissions of sample information.

BioSamples also synchronizes data with the NCBI BioSample database and imports data from ENA.

This document provides information about local installation and development environment setup for the BioSamples database.

== Table of contents

<>

<>

<>

<>

<>

<>

<>

== Softwares

* Git 2.17.1
* Java 17
* JDK 17
* Maven 3.6.3
* Docker 18.6

== Setup

[arabic]
. Run this in terminal to install the dependent software.
+
[source,sh]
----
sudo apt-get update
sudo apt-get install temurin-17-jdk maven git docker
----
. Verify the software versions.
+
[source,sh]
----
docker -v
# Docker version 18.06.1-ce

java -version
# openjdk version "17.0.12" 2024-07-16
----
. Install BioSamples on your computer.
+
[source,sh]
----
git clone https://github.com/EBIBioSamples/biosamples-v4.git
cd biosamples-v4
./mvnw -T 2C package
----
. Start BioSamples on your machine
+
[source,sh]
----
docker-compose up
----
+
If you get:
`ERROR: Couldn't connect to Docker daemon - you might need to run docker-machine start default`,
try:
`sudo docker-compose up`
. Access the web interface at http://localhost:8081/biosamples/. Initially, there is no data in the local instance.
. Create an ENA WEBIN account for API authentication and data upload
+
An ENA WEBIN account is required to upload data via the BioSamples API.
+
Production WEBIN accounts can be created here:
https://www.ebi.ac.uk/ena/submit/webin/auth/swagger-ui/index.html?configUrl=/ena/submit/webin/auth/v3/api-docs/swagger-config#/AdministrationAPI/createSubmissionAccount
+
To register a test account, replace `www` with `wwwdev` in the URL.

== Data import

An example JSON payload to POST to http://localhost:8081/biosamples/beta/samples can be found here:
https://github.com/EBIBioSamples/biosamples-v4/blob/master/models/core/src/test/resources/TEST1.json

=== NCBI

Download the XML dump (~400MB):

`wget http://ftp.ncbi.nih.gov/biosample/biosample_set.xml.gz`

Run the pipeline to submit the data to BioSamples API via REST:

`docker-compose up biosamples-pipelines-ncbi`

You may need to mount the directory where the XML file is located. Use a `docker-compose.override.yml` file to handle volume mounting.

=== MongoDB notes

A useful MongoDB client tool: http://www.mongoclient.com

== Developing

Docker can be run in a virtual machine (e.g., VirtualBox) if needed. You can mount shared folders for IDE use.

To build code changes:
`./mvnw -T 2C package`

To rebuild docker containers:
`docker-compose build`

To rebuild a single container:
`docker-compose build biosamples-pipelines`

To run a service with its dependencies:
`docker-compose up biosamples-webapp-api`

To run a containerized executable:
`docker-compose run --service-ports biosamples-pipelines`

To pass command-line arguments (note: replaces the default executable):
`docker-compose run --service-ports biosamples-pipelines java -jar pipelines-4.0.0-SNAPSHOT.jar --debug`

Monitoring and debugging info: http://www.jamasoftware.com/blog/monitoring-java-applications/

Combined Maven build and container launch:
`./mvnw -T 2C package && docker-compose build && docker-compose run --service-ports biosamples-pipelines`

If `docker-compose` is slow, check for large volumes in the source directory. Use these to clean up:

Remove all Docker volumes:
`docker volume ls -q | xargs -r docker volume rm`

Remove all Docker images:
`docker images -q | xargs -r docker rmi`

WARNING: The above removes everything Docker-related from your machine.

== Client usage

Add the spring-boot starter module for BioSamples in your Maven project:

....

    
        uk.ac.ebi.biosamples
        biosamples-spring-boot-starter
        5.3.7
    

....

....
maven {
  url 'https://gitlab.ebi.ac.uk/api/v4/projects/2669/packages/maven'
}
....

Configure `biosamples.client.uri` in your `application.properties` to point to the correct BioSamples instance.

== Issues and troubleshooting

=== Problems with spring-data-rest

Originally, Spring Data REST was used for exposing the API but had issues:

* Content type negotiation problems due to overlaps with Thymeleaf routes.
* Cannot serve XML even with converters.
* List ordering caused optional attributes to mix—better handled via `Map` of attributes.

=== Known issues

Solr has a limitation on field size (term vector length). Values over 255 characters are not indexed.

== License

link:https://github.com/EBIBioSamples/biosamples-v4/LICENSE[Apache 2.0]

Owner

Citation (citation.cff)

GitHub Events

Issues and Pull Requests

Last synced: 10 months ago