Laniakea Documentation¶

Laniakea provides the possibility to automate the creation of Galaxy-based virtualized environments through an easy setup procedure, providing an on-demand workspace ready to be used by life scientists and bioinformaticians.

Galaxy is a workflow manager adopted in many life science research environments in order to facilitate the interaction with bioinformatics tools and the handling of large quantities of biological data.

Once deployed each Galaxy instance will be fully customizable with tools and reference data and running in an insulated environment, thus providing a suitable platform for research, training and even clinical scenarios involving sensible data. Sensitive data requires the development and adoption of technologies and policies for data access, including e.g. a robust user authentication platform.

For more information on the Galaxy Project, please visit the https://galaxyproject.org

Laniakea has been developed by ELIXIR-IIB, the italian node of ELIXIR, within the INDIGO-DataCloud project (H2020-EINFRA-2014-2) which aims to develop PaaS based cloud solutions for e-science.

Note

Laniakea is in fast development. For this reason the code and the documentation may not always be in sync. We try to make our best to have good documentatation

Overview¶

Galaxy is a workflow manager adopted in many life science research environments in order to facilitate the interaction with bioinformatics tools and the handling of large quantities of biological data. Through a coherent work environment and an user-friendly web interface it organizes data, tools and workflows providing reproducibility, transparency and data sharing functionalities to users.

Currently, Galaxy instances can be deployed in three ways, each one with pros and cons: public servers, local servers and commercial cloud solutions. In particular, the demand for cloud solutions is rapidly growing (over 2400 Galaxy cloud servers launched in 2015, since they allowi the creation of a ready-to-use galaxy production environment avoiding initial configuration issues, requiring less technical expertise and outsourcing the hardware needs. Nevertheless relying on commercial cloud providers is quite costly and can pose ethical and legal drawbacks in terms of data privacy.

ELIXIR-IIB in the framework of the INDIGO-DataCloud project is developing a cloud Galaxy instance provider, allowing to fully customize each virtual instance through a user-friendly web interface, overcoming the limitations of others galaxy deployment solutions. In particular, our goal is to develop a PaaS architecture to automate the creation of Galaxy-based virtualized environments exploiting the software catalogue provided by the INDIGO-DataCloud community (www.indigo-datacloud.eu/service-component).

Once deployed each Galaxy instance will be fully customizable with tools and reference data and running in an insulated environment, thus providing a suitable platform for research, training and even clinical scenarios involving sensible data. Sensitive data requires the development and adoption of technologies and policies for data access, including e.g. a robust user authentication platform.

The system allows to setup and launch a virtual machines configured with the Operative System (CentOS 7 or Ubuntu 14.04/16.04) and the auxiliary applications needed to support a Galaxy production environment such as PostgreSQL, Nginx, uWSGI and Proftpd and to deploy the Galaxy platform itself. It is possible to choose between different tools preset, or flavors: basic Galaxy or Galaxy configured with a selection of tools for NGS analyses already installed and configured (e.g. SAMtools, BamTools, Bowtie, MACS, RSEM, etc…) together with reference data for many organisms.

Service architecture¶

The web front-end is designed to grant user friendly access to the service, allowing to easily configure and launch each Galaxy instance through the INDIGO FutureGateway portal.

Galaxy as a Cloud Service architecture

All the required components to automatically setup Galaxy instances (Galaxy and all its companion software) are deployed using the INDIGO PaaS Orchestrator and the Infrastructure Manager services, based on the TOSCA orchestration language. The service is compatible with both OpenNebula and OpenStack, its deployment on different e-infrastructures. Moreover, it supports both VMs and Docker containers, leaving the selection of the virtual environment to the service providers. This effectively removes the need to depend on particular configurations (e.g. OpenStack, OpenNebula or other private cloud solution like Amazon or Google).

Persistent storage is provided to store users and reference data and to install and run new (custom) tools and workflows. Data security and privacy are granted through the INDIGO ONEDATA component which, at the same time, allow for transparent access to the storage resources through token management. Data encryption implemented at file system level protects user’s data from any unauthorized access.

Automatic elasticity, provided using the INDIGO CLUES service component, enables dynamic cluster resources scaling, deploying and powering-on new working nodes depending on the workload of the cluster and powering-off them when no longer needed. This provides an efficient use of the resources, making them available only when really needed.

ELIXIR-IIB: The Italian Infrastructure for Bioinformatics¶

ELIXIR-IIB (elixir-italy.org) is the Italian Node of ELIXIR (elixir-europe.org) and collects most of the leading Italian institutions in the field of bioinformatics, including a vast and heterogeneous community of scientists that use, develop and maintain a large set of bioinformatics services. It represents the Italian Node of ELIXIR, an European research infrastructure which goal is to integrate research data from all over Europe and ensure a seamless service provision easily accessible by the scientific community.

ELIXIR-IIB is also one of the scientific communities providing use cases to the INDIGO-Datacloud project (H2020-EINFRA-2014-2) which aims to develop PaaS based cloud solutions for e-science.

For a complete overview of ELIXIR-IIB related projects and services, please visit: http://elixir-italy.org/en/

INDIGO-DataCloud¶

The INDIGO-DataCloud project (H2020-EINFRA-2014-2) aims to develop an open source computing and data platform, targeted at multi-disciplinary scientific communities, provisioned over public and private e-infrastructures.

In order to exploit the full capabilities of current cloud infrastructures, supporting complex workflows, data transfer and analysis scenarios, the INDIGO architecture is based on the analysis and the realization of use cases selected by different research communities in the areas of High Energy Physics, Bioinformatics, Astrophysics, Environmental modelling, Social sciences and others.

The INDIGO-DataCloud architecture

The INDIGO-DataCloud communities

INDIGO released two software release:

Release	Code name	URL
First release	MIDNIGHTBLUE	https://www.indigo-datacloud.eu/news/first-indigo-datacloud-software-release-out
Second release	ELECTRICINDIGO	https://www.indigo-datacloud.eu/news/electricindigo-second-indigo-datacloud-software-release

The INDIGO-DataCloud releases provide open source components for:

IaaS layer: increase the efficiency of existing Cloud infrastructures based on OpenStack or OpenNebula through advanced scheduling, flexible cloud/batch management, network orchestration and interfacing of high-level Cloud services to existing storage systems.
PaaS layer: easily port applications to public and private Clouds using open programmable interfaces, user-level containers, and standards-based languages to automate definition, composition and embodiment of complex set-ups.
Identity and Access Management: manage access and policies to distributed resources.
FutureGateway: a programmable scientific portal providing easy access to both the advanced PaaS features provided by the project and to already existing applications.
Data Management and Data Analytics Solutions: distribute and access data through multiple providers via virtual file systems and automated replication and caching.

For a complete list of INDIGO-DataCloud services, please visit: https://www.indigo-datacloud.eu/service-component

The ELIXIR-IIB use case in INDIGO¶

ELIXIR-IIB in the framework of the INDIGO-DataCloud project is developing a cloud Galaxy instance provider, allowing to fully customize each virtual instance through a user-friendly web interface, overcoming the limitations of others galaxy deployment solutions. In particular, our goal is to develop a PaaS architecture to automate the creation of Galaxy-based virtualized environments exploiting the software catalogue provided by the INDIGO-DataCloud community.

All Galaxy required components automatically deployed (INDIGO PaaS Orchestrator and the Infrastructure Manager):
- Galaxy
- PostgreSQL
- NGINX
- uWSGI
- Proftpd
- Galaxy tools (from ToolShed)
- Reference Data
User friendly access, allowing to easily configure and launch a Galaxy instance (INDIGO FutureGateway portal)
Authentication (Identity and Access Management and FutureGateway)
Persistent storage, data security and privacy (Onedata or IaaS block storage with filesystem encryption).
Cluster support with automatic elasticity (INDIGO CLUES).

ELIXIR-IIB use case architecture (single VM)

ELIXIR-IIB use case in INDIGO architecture for single Galaxy instances deployment.

ELIXIR-IIB use case archithecture (cluster)

ELIXIR-IIB use case in INDIGO architecture for Galaxy with cluster support deployment

References¶

INDIGO services: https://www.indigo-datacloud.eu/service-component

Get Galaxy Express¶

The Galaxy Express section allows user to deploy a standard Galaxy production environment.

The service instantiate a CentOS 7 Virtual Machine with Galaxy, all its companion software and tools already embedded. Once deployed each Galaxy instance can be further customized with tools and reference data.

See also

For a detailed descreption of all Web UI options see section: Laniakea options.

See also

To login into the portal see section: Authentication.

Instantiate Galaxy¶

Enter in the Galaxy Express section:
Describe your instance using the Instance description field, which will identfy your Galaxy in the job list, once your request is submitted.
Select the Instance flavor, (virtual CPUs and RAM):

Currently, the following pre-sets are available, but not all of them are enabled.

Name VCPUs RAM Total Disk Root Disk

small 1 2 GB 20 GB 20 GB

medium 2 4 GB 20 GB 20 GB

large 4 8 GB 20 GB 20 GB

xlarge 8 16 GB 20 GB 20 GB

xxlarge 16 32 GB 20 GB 20 GB
Copy & Paste your SSH key, to login in the Galaxy instance:
Storage section allows to select the user storage volume size. The Enable encryption flag is explained here: Get encrypted instance.
Select the Galaxy version, the instance administrator e-mail and your custom Galaxy:

Warning

Please insert a vail mail address. No check is performed on its syntax, but entering an incorrect email address will cause deployment failure if the encryption option is set.

Select Galaxy tools pre-set:
and reference dataset:
Finally, SUBMIT your request:

Galaxy login¶

The galaxy administrator password is automatically generated during the instatiation procedure and is the same for each deployed instance:

User: galaxy administrator e-mail

Password: galaxy_admin_password

Warning

The anonymous login is by default disabled.

Warning

Change Galaxy password and the API key as soon as possible!

Get Galaxy¶

The Galaxy section allows user to deploy a full Galaxy production environment.

The service allows to setup and launch a virtual machine configured with the Operative System CentOS 7 and the auxiliary applications needed to support a Galaxy production environment such as PostgreSQL, Nginx, uWSGI and Proftpd and to deploy the Galaxy platform itself and the selected Galaxt tools.

Warning

Everything is configured on the fly and, depending on the number of the tools to be installed may take time.

See also

For a detailed descreption of all Web UI options see section: Laniakea options.

See also

To login into the portal see section: Authentication.

Instantiate Galaxy¶

Enter the Galaxy section:
Describe your instance using the Instance description field, which will identfy your Galaxy in the job list, once your request is submitted.
Select your instance flavour (virtual CPUs and the memory size):

Currently, the following pre-sets are available, but not all of them are enabled.

Name VCPUs RAM Total Disk Root Disk

small 1 2 GB 20 GB 20 GB

medium 2 4 GB 20 GB 20 GB

large 4 8 GB 20 GB 20 GB

xlarge 8 16 GB 20 GB 20 GB

xxlarge 16 32 GB 20 GB 20 GB
Copy & Paste your SSH key, to login in the Galaxy instance:
Storage section allows to select the user storage volume size. The Enable encryption flag is explained here: Get encrypted instance.
Select the Galaxy version, the instance administrator e-mail and your custom Galaxy brand:

Warning

Please insert a vail mail address. No check is performed on its syntax, but entering an incorrect email address will cause deployment failure if the encryption option is set.

Select Galaxy tools pre-set:
and reference dataset:
Finally, SUBMIT your request:

Galaxy login¶

The galaxy administrator password and the API key are automatically generated during the instatiation procedure and are the same for each instance:

User: your user e-mail

Password: galaxy_admin_password

API key: ADMIN_API_KEY

Warning

The anonymous login is by default disabled.

Warning

Change Galaxy password and the API key as soon as possible!

Get encrypted instance¶

The service provides the possibility to encrypt the storage volume associated to the virtual machine on-demand.

Warning

Only the external volume, where Galaxy data are stored, is encrypted, not the Virtual Machine root disk.

To encypt the external volume storage flag encryption in Enable encryption box.

Cryptographic keys should never be transmitted in the clear. For this reason during Galaxy deployment user intervention is required.

Data privacy is granted through LUKS storage encryption: user will be required to insert a password to encrypt/decrypt data directly on the virtual instance during its deployment, avoiding any interaction with the cloud administrator(s).

An e-mail is sent to instance administrator the e-mail address configured in the Galaxy Configuration section.

Galaxy encryption mail configuration section

Warning

Make sure you have entered a valid mail address!

The e-mail is sent you only when the system is ready to accept your password and contains all the instructions to correctly encrypt/decrypt your system. The e-mail subject is [ELIXIR-ITALY] Laniakea storage encryption, sent by Laniakea@elixir-italy.org

Warning

If you don’t receive the e-mail:

Check you SPAM mail directory
Chek mail address spelling
Wait 15 minutes more.

Once the e-mail is arrived you can follow the step by step guide to encrypt your volume: Storage encryption procedure.

User is only asked to insert their alphanumeric password 3 times:

Set password
Confirm password
Open LUKS volume.

After the password injection the logout is automatic: the encryption procedure continues in background.

Default encryption algorithm¶

The default LUKS configuration are:

Cipher: aes-xts-plain64

Key size: 256 bit

Hash Algorithm for key derivation: sha256

See also

For a detailed description of all Web UI options see section: Laniakea options.

Get Galaxy cluster¶

Warning

Currently, this feature is under beta testing. Galaxy and tools are installed on-the-fly starting from a bare CentOS 7 image. The whole process, i.e. install Galaxy and tools, may take time. We will soon add the possibility to exploit images with tools to speed-up the configuration.

It is possible to deploy a Galaxy cluster, using SLURM as Resource Manager, with or without automatic elasticity support.

Galaxy cluster section: all working nodes are deployed with the Galaxy server and are always available.
Galaxy elastic cluster serciont: automatic elasticity enables dynamic cluster resources scaling, deploying and powering on new working nodes depending on the workload of the cluster and powering-off them when no longer needed. This provides an efficient use of the resources, making them available only when really needed.

The two sections provide the same configuration options.

Warning

Each node takes 12 minutes or more to be instantiated. Therefore, the job needs the same time to start. On the contrary if the node is already deployed the job will start immediately.

See also

For a detailed description Galaxy elastic cluster see section Cluster support (SLURM).

For a detailed descreption of all Web UI options see section: Laniakea options.

To login into the portal see section: Authentication.

Instantiate Galaxy¶

Enter in the Galaxy cluster section:
Describe your instance using the Instance description field, which will identfy your Galaxy in the job list, once your request is submitted.
Select the number of Virtual Worker Nondes of your Cluster:
Select the Instance flavor, (virtual CPUs and RAM) for your Front node, i.e. the Galaxy server, and for each Worker Node:

Currently, the following pre-sets are available:

Name VCPUs RAM Total Disk Root Disk

small 1 2 GB 20 GB 20 GB

medium 2 4 GB 20 GB 20 GB

large 4 8 GB 20 GB 20 GB

xlarge 8 16 GB 20 GB 20 GB

xxlarge 16 32 GB 20 GB 20 GB
Copy & Paste your SSH key, to login in the Galaxy instance:
Storage section allows to select the IaaS storage volume size. The Storage encryption option is explained here: Get encrypted instance.
Select the Galaxy version, the instance administrator e-mail and your custom Galaxy brand:

Warning

Please insert a vail mail address. No check is performed on its syntax, but entering an incorrect email address will cause deployment failure if the encryption option is set.

Select Galaxy tools pre-set:
and reference dataset:
Finally, SUBMIT your request:

Galaxy login¶

The galaxy administrator password and the API key are automatically generated during the instatiation procedure and are the same for each instance:

User: your user e-mail

Password: galaxy_admin_password

API key: ADMIN_API_KEY

Warning

The anonymous login is by default disabled.

Warning

Change Galaxy password and the API key as soon as possible!

Create SSH Keys¶

How to create SSH keys on Linux or macOS¶

https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/create-with-openssh/

How to create SSH keys on Windows¶

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/ssh-from-windows

Galaxy with Onedata¶

Warning

Galaxy with Onedata support is on-going and it is not currently available.

Virtual hardware presets¶

Each cloud provider enable a set of Image Flavor, defined in terms of Virctual CPUs (VCPUS), Memory, Disk, etc.

Currently, the following pre-sets are available:

Name	VCPUS	RAM	Total Disk	Root Disk
small	1	2 GB	20 GB	20 GB
medium	2	4 GB	20 GB	20 GB
large	4	8 GB	20 GB	20 GB
xlarge	8	16 GB	20 GB	20 GB
xxlarge	16	32 GB	20 GB	20 GB

New flavors can be assigned to particular projects.

Laniakea options¶

job identifier¶

Type: string

Description: Custom job identifier, it will identfy your Galaxy in the job list.

Mandatory: No

Virtual hardware configuration¶

Instance flavor¶

Type: selectable menu (string)

Description: Select instance flavor in terms of virtual CPUs, memory size (RAM) and disk size.

Mandatory: Yes

Sections: |galaxy_express|, Galaxy. Available as Galaxy server flavor on cluster configuration tabs.

Worker node number¶

Type: selectable menu (int)

Description: Maximum worker nodes number.

Mandatory: Yes

Sections: Galaxy cluster, Galaxy elastic cluster.

Worker nodes flavor¶

Type: selectable menu (string)

Description: Select worker nodee flavor in terms of virtual CPUs, memory size (RAM) and disk size.

Mandatory: Yes

Sections: Galaxy cluster, Galaxy elastic cluster.

SSH public key¶

Type: string

Description: User personale SSH public key. Required to login through SSH in the Galaxy VM

Mandatory: Yes

Sections: all.

Storage configuration¶

Storage type¶

Type: selectable menu (string)

Description: Select commont IaaS Block storage (IaaS) or Encrypted Storage (encryption).

Mandatory: Yes

Sections: all.

Storage size¶

Type: selectable menu (string)

Galaxy flavors¶

Type: selectable menu (string)

Description: Select tools presets to be installed on the server.

Mandatory: No

Sections: all.

Galaxy production environment¶

The system allows to setup and launch a virtual machine (VM) configured with the Operative System (CentOS 7 or Ubuntu 14.04/16.04) and the auxiliary applications needed to support a Galaxy production environment such as PostgreSQL, Nginx, uWSGI and Proftpd and to deploy the Galaxy platform itself. A common set of Reference data is available through a CernVM-FS volume.

Once deployed each Galaxy instance can be further customized with tools and reference data.

The Galaxy production environment is deployed according to Galaxy official documentation: https://docs.galaxyproject.org/en/latest/admin/production.html.

OS support¶

CentOS 7 is our default distribution, Given its adherence to Standards and the length of official support (CentOS-7 updates until June 30, 2024, https://wiki.centos.org/FAQ/General#head-fe8a0be91ee3e7dea812e8694491e1dde5b75e6d). CentOS 7 and Ubuntu (14.04 and 16.04) are both supported.

CentOS 7 and Ubuntu Xenial 16.04 exploit systemd as as init system, while Ubuntu Trusty 14.04 still uses upstart.

Warning

Selinux is by default disabled on CentOS.

PostgresSQL¶

PostgreSQL packages coming from PostgreSQL official repository are installed:

Distribution	Repository
Centos	https://wiki.postgresql.org/wiki/YUM_Installation
Ubuntu	https://wiki.postgresql.org/wiki/Apt

Current stable PostgreSQL version is installed: PostgreSQL 9.6

On CentOS 7 the default pgdata directory is /var/lib/pgsql/9.6/data. The pg_hba.conf configuration is modified allowing for password authentication. On CentOS we need to exclude CentOS base and updates repo for PostgreSQL, otherwise dependencies might resolve to the postgresql supplied by the base repository.

On Ubuntu default pgdata directory is /var/lib/postgresql/9.6/main, while the configuration files are stored in /etc/postgresql/9.6/main. There’s no need to modify the HBA configuration file since, by default, it is allowing password authentication.

PostgreSQL start/stop/status in entrusted to Systemd on CentOS 7 and Ubuntu Xenial and to Upstart for Ubuntu Trusty.

Distribution	Command
CentOS 7	sudo systemctl start/stop/status postgres-9.6
Ubuntu Xenial	sudo systemctl start/stop/status postgresql
Ubuntu Trusty	sudo service postgresql start/stop/status

Galaxy database configuration¶

Two different database are configured to track data and tool shed install data, allowing to bootstrap fresh Galaxy instance with pretested installs. The database passwords are randomly generated and the passoword can be retrieved in the galaxy.ini file.

Galaxy database is named galaxy and is configured in the galaxy.ini file:

database_connection = postgresql://galaxy:gtLxNnH7DpISmI5FXeeI@localhost:5432/galaxy

The shed install tool database is named galaxy_tools and is configured as:

install_database_connection = postgresql://galaxy:gtLxNnH7DpISmI5FXeeI@localhost:5432/galaxy_tools

Docker¶

On Docker container PostgreSQL cannot be managed through systemd/upstart, since there’s no init system on CentOS and Ubuntu docker images. Therefore, the system is automatically configured to run postgresql using supervisord.

NGINX¶

To improve Galaxy performance, NGINX is used as web server. The official Galaxy nginx packages are used by default (built in upload module support).

Distribution	Repository
Centos	https://depot.galaxyproject.org/yum/
Ubuntu	ppa:galaxyproject/nginx

Moreover, on Ubuntu, we need to prevent NGINX to be updated by apt default packages. For this purpose the pin priority of NGINX ppa packages is raised, by editing /etc/apt/preferences.d/galaxyproject-nginx-pin-700 (more on apt pinning at: https://wiki.debian.org/AptPreferences).

NGINX is configured following the official Galaxy wiki: https://galaxyproject.org/admin/config/nginx-proxy/.

NGINX is started, usually using the command line, from /usr/sbing/nginx:

$ sudo nginx

NGINX options¶

NGINX options are listed here: https://www.nginx.com/resources/wiki/start/topics/tutorials/commandline/

Option	Description
-?, -h	Print help.
-v	Print version.
-V	Print NGINX version, compiler version and configure parameters.
-t	Don’t run, just test the configuration file. NGINX checks configuration for correct syntax and then try to open files referred in configuration.
-q	Suppress non-error messages during configuration testing.
-s signal	Send signal to a master process: stop, quit, reopen, reload. (version >= 0.7.53)
-p prefix	Set prefix path (default: /usr/local/nginx/). (version >= 0.7.53)
-c filename	Specify which configuration file NGINX should use instead of the default.
-g directives	Set global directives. (version >= 0.7.4)

The main way to start/stop/reload nginx is through the -s command line option:

Action	Command
Start	sudo nginx
Stop	sudo nginx -s stop
Restart	First stop nginx then start it: `sudo nginx -s stop; sudo nginx`

Finally, to start/stop/status NGINX with systemd:

Dstribution	Command
CentOS 7	sudo systemctl start/stop/status nginx
Ubuntu Xenial	sudo systemctl start/stop/status nginx
Ubuntu Trusty	sudo service nginx start/stop/status

NGINX troubleshooting¶

Running NGINX on CentOS through systemd could lead to this error in /var/log/nginx/error.log, which can prevent Galaxy web page loading:

2017/08/24 08:22:32 [crit] 3320#0: *7 connect() to 127.0.0.1:4001 failed (13: Permission denied) while connecting to upstream, client: 192.167.91.214, server: localhost, request: "GET /galaxy HTTP/1.1", upstream: "uwsgi://127.0.0.1:4001", host: "90.147.102.159"

This is related to SELinux polixy on CentOS.

Warning

You should avoid to modify SELinux policy, since you can still use NGINX command line options.

Anyway, the problem is that selinux dany socket access. This results in a generic access denied error in NGINX’s log, the important messages are actually in selinux’s audit log. To solve this issue, you can ran the following commands as superuser.

# show the new rules to be generated
grep nginx /var/log/audit/audit.log | audit2allow

# show the full rules to be applied
grep nginx /var/log/audit/audit.log | audit2allow -m nginx

# generate the rules to be applied
grep nginx /var/log/audit/audit.log | audit2allow -M nginx

# apply the rules
semodule -i nginx.pp

Then restart NGINX.

You may need to generate the rules multiple times (likely four times to fix all policies), trying to access the site after each pass, since the first selinux error might not be the only one that can be generated.

Further readings

NGINX documentation: https://www.nginx.com/blog/nginx-se-linux-changes-upgrading-rhel-6-6/
StackOverflow post: https://stackoverflow.com/questions/26334526/nginx-cant-access-a-uwsgi-unix-socket-on-centos-7
Blog post: http://axilleas.me/en/blog/2013/selinux-policy-for-nginx-and-gitlab-unix-socket-in-fedora-19/

uWSGI¶

uWSGI (https://uwsgi-docs.readthedocs.io/en/latest) is used as interface between the web server (i.e. NGINX) and the web application (i.e. Galaxy). Using uWSGI for production servers is recommended by the Galaxy team: https://galaxyproject.org/admin/config/performance/scaling/

uWSGI configuration is embedded in the galaxy.ini file ($HOME/galaxy/config/galaxy.ini), with 4 handler configuration. By defalut the number of processes (i.e. uWSGI workers is set to number_of_virtual_cpus - 1. This configuration should be fine for most uses. Nevertheless, there’s no golden rule to define the workers number. It is up to the end-user to configure it dependig on your needs. The same goes for the number of job handlers (4 by default).

UWSGI socket and stats server are, by default, listening on 127.0.0.1:4001 and 127.0.0.1:9191, respectively. More on the uWSGI stats server here: http://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html?highlight=stats%20server.

UWSGI Galaxy Configuration:

[uwsgi]
master = True
processes = 1
socket = 127.0.0.1:4001
stats = 127.0.0.1:9191
pythonpath = /home/galaxy/galaxy/lib
pythonhome = /home/galaxy/galaxy/.venv
threads = 4
logto = /var/log/galaxy/uwsgi.log

# Job Handler(s)

[server:handler0]
use = egg:Paste@http
port = 8090
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler1]
use = egg:Paste@http
port = 8091
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler2]
use = egg:Paste@http
port = 8092
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler3]
use = egg:Paste@http
port = 8093
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

Proftpd¶

To allow user to upload files (> 2GB) through FTP, Proftpd is installed and configured on each Galaxy server, according to: https://galaxyproject.org/admin/config/upload-via-ftp/

Proftpd configuration file is located at /etc/proftdp.conf on CentOS and /etc/proftpd/proftpd.conf on Ubuntu.

To grant a user access to read emails and passwords from the Galaxy database, a separate user is created for the FTP server which has permission to SELECT from the galaxy_user table and nothing else.

Proftpd is listening on port 21. FTP protocol is not encrypted by default, thus any usernames and passwords are sent over clear text to Galaxy.

How to use FTP through FileZilla¶

You need to disable Passive (PASV) mode in FileZilla, since we are not going to open all passive ports.

Open FileZilla.
Click on Edit | Settings.
Open Connection menu on the left. Click on FTP menu.
Mark the Active radio button.
Click OK.

How to use FTP through command line¶

To install FTP command line client, type sudo yum install ftp on CentOS or sudo apt-get install ftp on Ubuntu.

To establish a connection with Glaxy Proftpd server, you can use your Galaxy username and password, in addition to the server IP address you’re connecting to (e.g. 90.147.102.82). To open a connection in Terminal type the following command, replacing the IP address with with your server IP address:

$ ftp 90.147.102.82
Connected to 90.147.102.82.
220 ProFTPD 1.3.5e Server (galaxy ftp server) [::ffff:90.147.102.82]
Name (90.147.102.82:marco):

Then login with your Galaxy credentials, typing your Galaxy e-mail address and password:

$ ftp 90.147.102.82
Connected to 90.147.102.82.
220 ProFTPD 1.3.5e Server (galaxy ftp server) [::ffff:90.147.102.82]
Name (90.147.102.82:marco): ma.tangaro@gmail.com
331 Password required for ma.tangaro@gmail.com
Password:

To upload file to your Galaxy remote directory:

ftp> put Sc_IP.fastq
local: Sc_IP.fastq remote: Sc_IP.fastq
229 Entering Extended Passive Mode (|||30023|)
150 Opening BINARY mode data connection for Sc_IP.fastq
8% |******                                                                           | 12544 KiB   23.84 KiB/s  1:31:23 ETA

Then you will find it on Galaxy:

Here’s a list of the basic commands that you can use with the FTP client.

Command	Description
ls	to find out the pathname of the current directory on the remote machine.
cd	to change directory on the remote machine.
pwd	to find out the pathname of the current directory on the remote machine.
delete	to delete (remove) a file in the current remote directory (same as rm in UNIX).
mkdir	to make a new directory within the current remote directory.
rmdir	to to remove (delete) a directory in the current remote directory.
get	to copy one file from the remote machine to the local machine
	`get ABC DEF` copies file ABC in the current remote directory to (or on top of) a file named DEF in your current local directory.
	`get ABC` copies file ABC in the current remote directory to (or on top of) a file with the same name, ABC, in your current local directory.
mget	to copy multiple files from the remote machine to the local machine; you are prompted for a y/n answer before transferring each file.
put	to copy one file from the local machine to the remote machine.
mput	o copy multiple files from the local machine to the remote machine; you are prompted for a y/n answer before transferring each file.
quit	to exit the FTP environment (same as bye).

Supervisord¶

Supervisor is a process manager written in Python, which allows its users to monitor and control processes on UNIX-like operating systems. It includes:

Supervisord daemon (privileged or unprivileged);
Supervisorctl command line interface;
INI config format;
[program:x] defines a program to control.

Supervisord requires root privileges to run.

Galaxy supervisord configuration is located here: https://docs.galaxyproject.org/en/master/admin/framework_dependencies.html?highlight=uwsgi#supervisor

and here: https://galaxyproject.github.io/dagobah-training/2016-saltlakecity/002a-systemd-supervisor/systemd-supervisor.html#1

A configuration running the Galaxy server under uWSGI has been installed on /etc/supervisord.d/galaxy_web.ini on CentOS, while it is located on /etc/supervisor/conf.d/galaxy.conf on Ubuntu. The options stopasgroup = true and killasgroup = true ensure that the SIGINT signal, to shutdown Galaxy, is propagated to all uWSGI child processes (i.e. to all uWSGI workers).

PYTHONPATH is not specified in this configuration since it was conflicting with Conda running.

To manage Galaxy through supervisord:

Action	Command
Start Galaxy	sudo supervisorctl start galaxy:
Stop Galaxy	sudo supervisorctl stop galaxy:
Restart Galaxy	sudo supervisorctl restart galaxy:
Galaxy status	sudo supervisorctl status galaxy:

$ supervisorctl help

default commands (type help <topic>):
=====================================
add    clear  fg        open  quit    remove  restart   start   stop  update
avail  exit   maintail  pid   reload  reread  shutdown  status  tail  version

$ sudo supervisorctl status galaxy:
galaxy:galaxy_web                RUNNING   pid 9030, uptime 2 days, 21:19:28
galaxy:handler0                  RUNNING   pid 9031, uptime 2 days, 21:19:28
galaxy:handler1                  RUNNING   pid 9041, uptime 2 days, 21:19:27
galaxy:handler2                  RUNNING   pid 9046, uptime 2 days, 21:19:26
galaxy:handler3                  RUNNING   pid 9055, uptime 2 days, 21:19:25

galaxy_web.ini file configuration:

[program:galaxy_web]
command         = /home/galaxy/galaxy/.venv/bin/uwsgi --virtualenv /home/galaxy/galaxy/.venv --ini-paste /home/galaxy/galaxy/config/galaxy.ini --pidfile /var/log/galaxy/uwsgi.pid
directory       = /home/galaxy/galaxy
umask           = 022
autostart       = true
autorestart     = true
startsecs       = 20
user            = galaxy
environment     = PATH="/home/galaxy/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
numprocs        = 1
stopsignal      = INT
startretries    = 15
stopasgroup     = true
killasgroup     = true

[program:handler]
command         = /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler%(process_num)s --log-file=/var/log/galaxy/handler%(process_num)s.log
directory       = /home/galaxy/galaxy
process_name    = handler%(process_num)s
numprocs        = 4
umask           = 022
autostart       = true
autorestart     = true
startsecs       = 20
user            = galaxy
startretries    = 15

[group:galaxy]
programs = handler, galaxy_web

Finally, a systemd script has been installed to start/stop Supervisord on /etc/systemd/system/supervisord.service.

Action	Command
Start	sudo systemctl start supervisord.service
Stop	sudo systemctl stop supervisord.service
Restart	sudo systemctl restart supervisord.service
Status	sudo systemctl status supervisord.service

$ sudo systemctl status supervisord.service
  ● supervisord.service - Supervisor process control system for UNIX
   Loaded: loaded (/etc/systemd/system/supervisord.service; disabled; vendor preset: disabled)
   Active: active (running) since Sat 2017-08-12 08:48:33 UTC; 9s ago
     Docs: http://supervisord.org
 Main PID: 12204 (supervisord)
   CGroup: /system.slice/supervisord.service
           ├─12204 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisord.conf
           ├─12207 /home/galaxy/galaxy/.venv/bin/uwsgi --virtualenv /home/galaxy/galaxy/.venv --ini-paste /home/galaxy/galaxy/config/galaxy.ini --pidfile /var/log/galaxy/uwsgi.pid
           ├─12208 /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler0 --log-file=/var/log/galaxy/handler0.log
           ├─12209 /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler1 --log-file=/var/log/galaxy/handler1.log
           ├─12210 /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler2 --log-file=/var/log/galaxy/handler2.log
           └─12211 /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler3 --log-file=/var/log/galaxy/handler3.log

Aug 12 08:48:33 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:33,805 CRIT Supervisor running as root (no user in config file)
Aug 12 08:48:33 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:33,805 WARN Included extra file "/etc/supervisord.d/galaxy_web.ini" during parsing
Aug 12 08:48:34 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:34,564 INFO RPC interface 'supervisor' initialized
Aug 12 08:48:34 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:34,564 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Aug 12 08:48:34 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:34,565 INFO supervisord started with pid 12204
Aug 12 08:48:35 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:35,569 INFO spawned: 'galaxy_web' with pid 12207
Aug 12 08:48:35 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:35,573 INFO spawned: 'handler0' with pid 12208
Aug 12 08:48:35 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:35,576 INFO spawned: 'handler1' with pid 12209
Aug 12 08:48:35 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:35,581 INFO spawned: 'handler2' with pid 12210
Aug 12 08:48:35 galaxy-indigo-test supervisord[12204]: 2017-08-12 08:48:35,584 INFO spawned: 'handler3' with pid 12211

Galaxy init scripts¶

Systemctl is the command line interface to systemd:

systemctl <start|stop|restart|...> <name>[.service]
systemctl <enable|disable> <name>[.service]

Since CentOS and Ubuntu Xenial 16.04 exploits systemd as init system, the Galaxy init script is located in /etc/systemd/system/galaxy.service.

Action	Command
Start	sudo systemctl start galaxy.service
Stop	sudo systemctl stop galaxy.service
Restart	sudo systemctl restart galaxy.service
Status	sudo systemctl status galaxy.service

Ubuntu Trusty 14.04 exploits Upstart as init system. Galaxy init file is located in /etc/init.d/galaxy.

Action	Command
Start	sudo service galaxy start
Stop	sudo service galaxy stop
Restart	sudo service galaxy restart
Status	sudo service galaxy status

Galaxy instance isolation¶

User data are automatically stored to the “/export” directory, where an external (standard block storage) volume is mounted.

All Galaxy job results are stored in this directory through galaxy.ini configuration file. For instance, files directory is located:

# Dataset files are stored in this directory.
file_path = /export/galaxy/database/files

while the job working directory is located:

# Each job is given a unique empty directory as its current working directory.
# This option defines in what parent directory those directories will be
# created.
job_working_directory = /export/job_work_dir

Here is the list of Galaxy database path directories:

file_path = /export/galaxy/database/files
job_working_directory = /export/job_work_dir
new_file_path = /export/galaxy/database/tmp
template_cache_path = /export/galaxy/database/compiled_templates
citation_cache_data_dir = /export/galaxy/database/citations/data
citation_cache_lock_dir = /export/galaxy/database/citations/lock
whoosh_index_dir = /export/galaxy/database/whoosh_indexes
object_store_cache_path = /export/galaxy/database/object_store_cache
cluster_file_directory = /export/galaxy/database/pbs"
ftp_upload_dir = /export/galaxy/database/ftp

The service offers the possibility to store data exploitng file system encryption or Onedata.

Using file system encryption each Galaxy instance can be deployed as an insulated environment, i.e. data are isolated from any other instance on the same platform and from the cloud service administrators, opening to the adoption of Galaxy based cloud solutions even within clinical environments. Data privacy is granted through LUKS file system encryption: users will be required to insert a password to encrypt/decrypt data directly on the virtual instance during its deployment, avoiding any interaction with the cloud administrator(s). The encryption procedure is described on Storage encryption procedure.

Alternatively, Users’ data access rights will be controlled through the OneData INDIGO component: Onedata (beta)

Storage encryption¶

While the adoption of a distributed environment for data analysis makes data difficult to be tracked and identified by a malevolus attacker, full data anonymity and isolation is still not granted.

For this reason, we have introduced the possibility for the user to isolate data through standard filesystem encryption, using the Linux device mapper encryption module, dm-crypt as encryption backend, cryptsetup, which is a command line interface to dm-crypt and LUKS (Linux Unified Key Setup) as encryption mode.

Disk encryption ensures that files are stored on disk in an encrypted form: the files only become available to the operating system and applications in readable when the volume is unlocked by a trusted user. The adopted block device encryption method, operates below the filesystem layer and ensures that everything is written to the block device (i.e. the external volume) is encrypted.

Dm-crypt is the standard device-mapper encryption functionality provided by the Linux kernel. Dm-crypt management is entrusted to the cryptsetup userspace utility, using LUKS as default block-device encryption method. LUKS, is an additional layer which stores all the needed setup information for dm-ctypt on the disk itself, abstracts partition and key management in an attempt to improve ease of use and cryptographic security

To automatically setup LUKS volumes on your Galaxy instances a bash script, named fast-luks has been created. The script has been integrated in the Galaxy instantiation procedure: if the File System Encryption option is selected through the dialogue window the users will be required to insert a password to encrypt/decrypt data on the virtual instance during its deployment, avoiding any interaction with the cloud administrator(s). For more details see: Fast-luks script.

Note

During the ancryption procedure an e-mail is sent to the Galaxy Instance Administrator, with the Virtual Machine IP address and a detailed description of the procedure to inject the encryption password in the system. The Galaxy installation procedure is paused until the password is correctly set (after 5 hours the system will return an error).

Warning

The system does not store your keys on its servers and cannot access your protected data unless you provide the key. This also means that if you forget or lose your key, there is no way to recover the key or to recover any data encrypted with the lost key.

Default configuration¶

Fast-luks, by default adopt xts-aes-plain64 cipher with 256 bit keys ans sha256 hashing algorithm.

Once the LUKS partition is created, it is unlocked.

The unlocking process will map the partition to a new device name using the device mapper. This alerts the kernel that device is actually an encrypted device and should be addressed through LUKS using the /dev/mapper/<cryptdev_name> so as not to overwrite the encrypted data. cryptdev_name is random generated to avoid accidental overwriting.

The volume is mounted, by default, on /export, with standard ext4 filesystem and Galaxy is configured to store here datasets.

Defaults values¶

# Defaults
cipher_algorithm='aes-xts-plain64'
keysize='256'
hash_algorithm='sha256'
device='/dev/vdb'
cryptdev='crypt'
mountpoint='/export'
filesystem='ext4'

Password choice¶

During the encryption procedure ( Storage encryption procedure), users are required to configure their encryption password. You must provide at least an alphanumeric 8-character long password. A random generated password is generated by the script, ONLY FOR EXAMPLE!

During the encryption procedure, the password will be required three times.

The script will query for passwords twice, to verify it. Then, you have to unlock your device, by inserting your password.

The procedure is described here: ( Storage encryption procedure).

Luksctl¶

During the encryption procedure ( Storage encryption procedure), fast-luks creates a configuration ini file, allowing Galaxy administrator to easily mange LUKS devices, through the luksctl script (Luksctl: LUKS volumes management).

By default the file is stored in /etc/galaxy/luks-cryptdev.ini.

The script requires superuser rights.

Action	Command	Description
Open	sudo luksctl open	Open the encrypted device, requiring your passphrase.
Close	sudo luksctl close	Close the encrypted device
Status	sudo luksctl status	Check device status using dm-setup

Galaxyctl can be used to parse luksctl commands:

Action	Command	Description
Open LUKS volume	sudo galaxyctl open luks	Open the encrypted device, requiring your passphrase.
Close LUKS volume	sudo galaxyctl close luks	Close the encrypted device
Check LUKS volume	sudo galaxyctl status luks	Check device status using dm-setup

Fast-luks script¶

The fast-luks script is located in /usr/local/bin/fast-luks.

It parse common cryptsetup parameters to encrypt the volume. For this reason it checks for cryptsetup and dm-setup packages and it install cryptsetup, if not installed.

Typing sudo fast-luks the script will load defaults parameters and will LUKS format /dev/vdb device, otherwise different parameters can be specified.

NB: Run as root.

Argument	Defaults	Description
`-c`, `--cipher`	aes-xts-plain64	Set cipher specification string.
`-k`, `--keysize`	256	Set key size in bits.
`-a`, `--hash_algorithm`	sha256	For luksFormat action specifies hash usedin LUKS key setup scheme and volumekey digest.
`-d`, `--device`	/dev/vdb	Set device to be mounted
`-e`, `--cryptdev`	crypt	Sets up a mapping <name> after successfulverification of the supplied key(via prompting).
`-m`, `--mountpoint`	/export	Set mount point
`-f`, `--filesystem`	ext4	Set filesystem

$ sudo fast-luks --help
=========================================================
                      ELIXIR-Italy
               Filesystem encryption script

A password with at least 8 alphanumeric string is needed
There's no way to recover your password.
Example (automatic random generated passphrase):
                      PcHhaWx4

You will be required to insert your password 3 times:
  1. Enter passphrase
  2. Verify passphrase
  3. Unlock your volume

The connection will be  automatically closed.

=========================================================

fast-luks: a bash script to automate LUKS file system encryption.
 usage: fast-luks [-h]

 optionals argumets:
 -h, --help           show this help text
 -c, --cipher                 set cipher algorithm [default: aes-xts-plain64]
 -k, --keysize                set key size [default: 256]
 -a, --hash_algorithm         set hash algorithm used for key derivation
 -d, --device                 set device [default: /dev/vdb]
 -e, --cryptdev               set crypt device [default: cryptdev]
 -m, --mountpoint             set mount point [default: /export]
 -f, --filesystem             set filesystem [default: ext4]
 --default                    load default values

Cryptsetup howto¶

The cryptsetup action to set up a new dm-crypt device in LUKS encryption mode is luksFormat:

cryptsetup -v --cipher aes-xts-plain64 --key-size 256 --hash sha 256 --iter-time 2000 --use-urandom --verify-passphrase luksFormat crypt --batch-mode

where crypt is the new device located to /dev/mapper/crypt.

To open and mount to /export an encrypted device:

cryptsetup luksOpen /dev/vdb crypt

mount /dev/mapper/crypt /export

To show LUKS device info:

dmsetup info /dev/mapper/crypt

To umount and close an encrypted device:

umount /export

cryptsetup close crypt

To force LUKS volume removal:

dmsetup remove /dev/mapper/crypt

..Note:

NB: Run as root.

Change LUKS password¶

LUKS provides 8 slots for passwords or key files. First, check, which of them are used:

cryptsetup luksDump /dev/<device> | grep Slot

where the output, for example, looks like:

Key Slot 0: ENABLED
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Then you can add, change or delete chosen keys:

cryptsetup luksAddKey /dev/<device> (/path/to/<additionalkeyfile>)

cryptsetup luksChangeKey /dev/<device> -S 6

As for deleting keys, you have 2 options:

delete any key that matches your entered password:
```
cryptsetup luksRemoveKey /dev/<device>
```

delete a key in specified slot:

cryptsetup luksKillSlot /dev/<device> 6

References¶

Disk encryption archlinux wiki page: https://wiki.archlinux.org/index.php/disk_encryption#Block_device_encryption_specific

Dm-crypt archlinux wiki page: https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption#Encryption_options_for_LUKS_mode

Original LUKS script: https://github.com/JohnTroony/LUKS-OPs/blob/master/luks-ops.sh (Credits to John Troon for the original script))

LUKS: https://guardianproject.info/code/luks/

LUKS how-to: http://www.thegeekstuff.com/2016/03/cryptsetup-lukskey

Storage encryption procedure¶

To encrypt the Virtual machine external volume follow this procedure.

Virtual Machine login¶

Log-in into your machine with:

ssh -i your_private_ssh_key.key galaxy@virtual.machine.ip.address

Typical IP addresses are: 90.147.170.xx, 90.147.102.xx or 90.147.75.xx and it is reported in the e-mail we sent you. You can copy and past the command from the mail the system send you.

Probably, you have to permanently accept the connection, typing “yes”.

and then enter your SSH passphrase.

Passphrase creation¶

You will be now prompted in the encryption script automatically. You will be required to insert an alphanumeric key, at least 8 characters. A key is automatically generated, as example, plase do not use if for production!

You have to type your password three times:

inject your password
Confirm your password
Unlock your encrypted volume

Insert your volume encrypt/decrypt password for the first time:

and confirm it:

If the passphrases don’t match, restart the procedure.

Unlock the volume¶

Unlock the encrypted volume typing again your password:

The volume will be now encrypted and you will be automatically log-out the VM, until Galaxy is installed.

File System Encryption Test¶

Test executed to ensure LUKS volume encryption.

Create two volumes, here named vol1, vol2.

Attach each one to the instance (here listed as /dev/vdd and /dev/vde) and mount them respectively to /export and /export1.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/vdd        976M  2.6M  907M   1% /export
/dev/vde        976M  2.6M  907M   1% /export1

Encrypt /export, i.e. /dev/vdd using fast_luks (/export is the default value).

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/vde              976M  2.6M  907M   1% /export1
/dev/mapper/jtedehex  990M  2.6M  921M   1% /export

Ensure that /export has the same permissions of the other two volumes.

drwxr-xr-x.   3 centos centos 4096 Nov  9 10:27 export
drwxr-xr-x.   3 centos centos 4096 Nov  9 10:27 export1

Put the same file on both volumes:

$ echo "encryption test" > /export/test.txt
$ echo "encryption test" > /export1/test.txt

Umount all the volumes and luksClose the encrypted one:
```
$ sudo cryptsetup luksClose /dev/mapper/jtedehex
```

Create the volume binary image using dd:

sudo dd if=/dev/vdd of=/home/centos/vdd_out
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 21.809 s, 49.2 MB/s

$ sudo dd if=/dev/vde of=/home/centos/vde_out
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 21.3385 s, 50.3 MB/s

HexDump the binary image with xdd:

$ xxd vdd_out > vdd.txt

$ xxd vde_out > vde.txt

As output you should have:

$ ls -ltrh
-rw-r--r--.  1 root   root   1.0G Nov  9 11:19 vdd_out
-rw-r--r--.  1 root   root   1.0G Nov  9 11:22 vde_out
-rw-rw-r--.  1 centos centos 4.2G Nov  9 11:32 vdd.txt
-rw-rw-r--.  1 centos centos 4.2G Nov  9 11:36 vde.txt

Grep non-zero bytes and search for the test.txt file content encryption test:

$ grep -v "0000 0000 0000 0000 0000 0000 0000 0000" vde.txt > grep_vde.txt
$ grep "encryption test" grep_vde.txt
8081000: 656e 6372 7970 7469 6f6e 2074 6573 740a  encryption test.

$ grep -v "0000 0000 0000 0000 0000 0000 0000 0000" vdd.txt > grep_vdd.txt
$ grep "encryption test" grep_vdd.txt
$

Note

It is possible to see the test.txt file content only on the un-encrypted volume.

Moreover, the output file grep_vde.txt is 73 kb while the encrypted one, grep_vdd.txt (138 MB), is very large:

-rw-rw-r--.  1 centos centos  73K Nov  9 11:46 grep_vde.txt
-rw-rw-r--.  1 centos centos 138M Nov  9 11:58 grep_vdd.txt

We also tried to open the volume when active (LUKS volume opened and mounted, Galaxy running) in the Virtual Machine, using the cloud controller (as administrator).

Test executed on the cloud controller:

# rbd map volume-3bedc7bc-eaed-466f-9d55-f2c29b44a7b2 --pool volumes
/dev/rbd0

# lsblk -f
NAME          FSTYPE      LABEL UUID                                   MOUNTPOINT
sda
|-sda1        ext4              db06fc46-7231-4189-ba2b-0b0117049680   /boot
|-sda2
|-sda5        swap              e5b98538-8337-4e25-8f82-f97f04258716   [SWAP]
`-sda6        LVM2_member       n4SAgY-GRNy-4Fl2-ROoQ-rRIf-bdBP-QC1B6s
  `-vg00-root ext4              1e3f1ff1-8677-4236-8cb4-07d5cad32441   /
rbd0          crypto_LUKS       c4bee3b9-e0dc-438e-87ae-2a3e491081c0

# mount /dev/rbd0 /mnt/
mount: unknown filesystem type ‘crypto_LUKS’

It is not possible to mount the volume without the user password.

Onedata (beta)¶

Transparent access to the storage resources through token management (Onedata). Each Galaxy instance uses of OneData spaces to store data.

Get Onedata space support¶

Galaxy ShedTools¶

Each Galaxy instance is customizable, through the web front-end, with different sets of pre installed tools (e.g. SAMtools, BamTools, Bowtie, MACS, RSEM, etc…), exploiting CONDA as default dependency resolver.

New Tools automatically installed using the official GalaxyProject python library Ephemeris.

Current possible pre-sets:

galaxy-epigen: Galaxy ready for NGS analysis.
galaxy-rna-workbench: Galaxy ready for RNA Sequencing analysis (original galaxy flavor here).
galaxy-testing: Galaxy test recipe.

The corresponding recipe on github is downloaded and processed. It is possible to easily add new flavors, just adding new recipes on github.

Create and test Galaxy flavors¶

For each tool you want to install, you must provide tool name and owner and one between tool_panel_section_id and tool_panel_section_label.

Keys	Required	Default value	Description
`name`	yes		This is is the name of the tool to install
`owner`	yes		Owner of the Tool Shed repository from where the tools is being installed
`tool_panel_section_id`	yes, if `tool_panel_section_label` not specified		ID of the tool panel section where you want the tool to be installed. The section ID can be found in Galaxy’s `shed_tool_conf.xml` config file. Note that the specified section must exist in this file. Otherwise, the tool will be installed outside any section.
`tool_panel_section_label`	yes, if `tool_panel_section_id` not specified		Display label of a tool panel section where you want the tool to be installed. If it does not exist, this section will be created on the target Galaxy instance (note that this is different than when using the ID). Multi-word labels need to be placed in quotes. Each label will have a corresponding ID created; the ID will be an all lowercase version of the label, with multiple words joined with underscores (e.g., ‘BED tools’ -> ‘bed_tools’).
`tool_shed_url`		`https://toolshed.g2.bx.psu.edu)`	The URL of the Tool Shed from where the tool should be installed.
`revisions`		latest	A list of revisions of the tool, all of which will attempt to be installed.
`install_tool_dependencies`		True	True or False - whether to install tool dependencies or not.
`install_repository_dependencies`		True	True or False - whether to install repo dependencies or not, using classic toolshed packages

For instance, this is the galaxy TESTING recipe:

---
api_key: <Admin user API key from galaxy_instance>
galaxy_instance: <Galaxy instance IP>

tools:
- name: fastqc
  owner: devteam
  tool_panel_section_label: 'Tools'
  install_resolver_dependencies: True
  install_tool_dependencies: False

- name: 'bowtie_wrappers'
  owner: 'devteam'
  tool_panel_section_label: 'Tools'
  install_resolver_dependencies: True
  install_tool_dependencies: False

Conda support¶

Conda is a package manager like apt-get, yum, pip, brew or guix and it is, currently, used as default dependency resolver in Galaxy.

References¶

Galaxy flavors: https://github.com/bgruening/docker-galaxy-stable#Extending-the-Docker-Image

Ephemeris: https://ephemeris.readthedocs.io/en/latest/

Ephemeris documentation: https://github.com/galaxyproject/ephemeris

Conda for Galaxy tools dependencies: https://docs.galaxyproject.org/en/master/admin/conda_faq.html

Reference Data¶

Many Galaxy tools rely on the presence of reference data, such as alignment indexes or reference genome sequences, to efficiently work. A complete set of Reference Data, able to work with most common tools for NGS analysis is available for each Galaxy instance deployed.

Available reference data¶

Reference data (e.g. genomic sequences) are available for many species and shared among all the instances, avoiding unnecessary and costly data duplication, exploiting the CernVM-FS filesystem or Onedata.

Until now, Galaxy administrators have been responsible for downloading, building and installing these important reference data. For example, to make the UCSC hg19 build of the human reference genome available to the Burrows-Wheeler Aligner (BWA) short-read mapper (Li and Durbin, 2009), a Galaxy administrator would need to (i) download the reference genome FASTA file, (ii) make it available as a reference genome via the ‘all_fasta’ table (optional), (iii) build BWA alignment indexes via proper command-line calls, (iv) register the location and availability of the indexes within the ‘bwa_indexes’ data table (by adding an additional entry to the tool-data/bwa_index.loc file on disk) and (v) finally, restart the Galaxy server.

A complete list of the reference data, with download link, is available here.

Arabidopis thaliana (TAIR9)
Arabidopis thaliana (TAIR10)
Drosophila melanogaster (dm3)
Homo sapiens (hg18)
Homo sapiens (hg19)
Homo sapiens (hg38)
Mus musculus (mm9)
Mus musculus (mm10)
Saccharomyces cerevisiae (sacCer3)

Reference data indexes available for bowite

Homo Sapiens - hg19¶

Available reference index	Details
bowtie
bowtie2
bwa
liftover
rsem	created with rsem v1.3.0, table and gtf file from ucsc (download date 11 March 2018)
.fa
.2bit
.gtf	download date: 11 March 2018

CernVM-FS reference data¶

The CernVM-File System (conversely cvmfs) provides a scalable, reliable and low- maintenance software distribution service. It was developed to assist High Energy Physics (HEP) collaborations to deploy software on the worldwide- distributed computing infrastructure used to run data processing applications.

CernVM-FS is implemented as a POSIX read-only file system in user space (a FUSE module). The reference data Files and directories are hosted on standard web servers and mounted on /refdata directory:

$ ls -l /refdata/elixir-italy.galaxy.refdata/
total 60
drwxr-xr-x. 5 cvmfs cvmfs 4096 May 21 20:10 at10
drwxr-xr-x. 5 cvmfs cvmfs 4096 May 21 20:10 at9
drwxr-xr-x. 3 cvmfs cvmfs 4096 May 21 20:10 dm2
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 20:11 dm3
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 20:15 hg18
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 18:36 hg19
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 20:18 hg38
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 20:22 mm10
drwxr-xr-x. 3 cvmfs cvmfs 4096 May 21 20:22 mm8
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 20:25 mm9
-rw-r--r--. 1 cvmfs cvmfs   57 May 21 18:31 new_repository
drwxr-xr-x. 3 cvmfs cvmfs 4096 May 21 20:25 sacCer1
drwxr-xr-x. 3 cvmfs cvmfs 4096 May 21 20:25 sacCer2
drwxr-xr-x. 7 cvmfs cvmfs 4096 May 21 20:25 sacCer3
-rw-r--r--. 1 cvmfs cvmfs    0 May 21 18:31 test-content

Cvmfs client setup¶

Cvmfs is installed by default on each Galaxy instance (CentOS 7 or Ubuntu 16.04) if this is the reference data configuration provided by your service provider.

The elixir-italy.galaxy.refdata.pub public key is installed in /etc/cvmfs/keys/. The /etc/cvmfs/default.local file is also already configured.

The cvmfs_config probe command mount the cvmfs volume to /cvmfs, therefore mount command is used to correctly mount the cvmfs volume to /refdata.

Description	Command
check configuration	cvmfs_config chksetup
mount volume	mount -t cvmfs elixir-italy.galaxy.refdata /refdata/elixir-italy.galaxy.refdata
umount volume	cvmfs_config umount elixir-italy.galaxy.refdata
reload repository	cvmfs_config reload elixir-italy.galaxy.refdata

Note

If mount fails, try to restart autofs with sudo service autofs restart.

Note

Cvmfs commands require root privileges

Cvmfs mount output:

$ sudo mount -t cvmfs elixir-italy.galaxy.refdata /refdata/elixir-italy.galaxy.refdata
CernVM-FS: running with credentials 994:990
CernVM-FS: loading Fuse module... done

$ ls /refdata/elixir-italy.galaxy.refdata/
at10  at9  dm2  dm3  hg18  hg19  hg38  mm10  mm8  mm9  new_repository  sacCer1  sacCer2  sacCer3  test-content

Cvmfs server location¶

Current cvmfs server configuration:

Reference data cvmfs	Details
cvmfs repository name	elixir-italy.galaxy.refdata
cvmfs server url	90.147.102.186
cvmfs key file	elixir-italy.galaxy.refdata.pub
cvmfs proxy url	DIRECT

Troubleshooting¶

Cvmfs not running, e.g. after reboot:

$ sudo mount -t cvmfs elixir-italy.galaxy.refdata /refdata/elixir-italy.galaxy.refdata
CernVM-FS: running with credentials 994:990
CernVM-FS: loading Fuse module... Failed to initialize root file catalog (16 - file catalog failure)

A reload of the config is able to fix the problem: https://wiki.chipp.ch/twiki/bin/view/CmsTier3/IssueCvmfsFailsToMount

$ sudo cvmfs_config reload elixir-italy.galaxy.refdata
Connecting to CernVM-FS loader... done
Entering maintenance mode
Draining out kernel caches (60s)
Blocking new file system calls
Waiting for active file system calls
Saving inode tracker
Saving chunk tables
Saving inode generation
Saving open files counter
Unloading Fuse module
Re-Loading Fuse module
Restoring inode tracker...  done
Restoring chunk tables...  done
Restoring inode generation...  done
Restoring open files counter...  done
Releasing saved glue buffer
Releasing chunk tables
Releasing saved inode generation info
Releasing open files counter
Activating Fuse module

Cvmfs server details¶

Since, cvmfs relies on OverlayFS or AUFS as default storage driver and Ubuntu 16.04 natively supports OverlayFS, it is used as default choice to create and populate the cvmfs server.

A resign script is located in /usr/local/bin/Cvmfs-stratum0-resign and the corresponding weekly cron job is set to /etc/cron.d/cvmfs_server_resign.

Log file is located in /var/log/Cvmfs-stratum0-resign.log.

Cvmfs references¶

CernVM-FS: https://cernvm.cern.ch/portal/filesystem

Cvmfs documentation: http://cvmfs.readthedocs.io/en/stable/

Onedata reference data (beta)¶

To Be Updated

Reference data local download¶

The reference data set can be downloaded on your machine. This option is not explicitly available, by default, on the service web intereface to avoid unuseful replication and costly virtual space consumption.

Nevertheless, it is still possible to download them through ansible and automatically configure galaxy to use them: Galaxycloud-refdata.

Cluster support (SLURM)¶

The service provides support for virtual clusters through a dedicated section of the web front-end and allows to instantiate Galaxy with SLURM (slurm.schedmd.com) as Resource Manager and to customize the number of virtual nodes, nodes and master virtual hardware.

Automatic elasticity, provided using the CLUES INDIGO service component (INDIGO CLUES), enables dynamic cluster resources scaling, deploying and powering on new working nodes depending on the workload of the cluster and powering-off them when no longer needed. This provides an efficient use of the resources, making them available only when really needed.

Each node is configured according to the Galaxy tools installed on the VM as selected by the user during the configuration phase. All tools pre-set are tested to work with the galaxy elastic cluster.

Conda packages used to solve Galaxy tools dependencies are stored in /export/_conda directory and shared between front and worker nodes. This

The service is scalable and both users and service providers can chose among a full range of different computational capabilities: from limited ones to serve e.g. small research groups, Galaxy developers or for didactic and training purposes, to instances with elasticity cluster support to deliver enough computational power for any kind of bioinformatic analysis and number of users, opening the route for the migration of public Galaxy instances to this service.

Shared file system¶

Current cluster configuration foresee two path shared between front and worker nodes:

/home where galaxy is installed.
/export where galaxy input and output datasets are hosted.

Note

NFS exports configuration file: /etc/exports

Nodes deployment¶

Warning

Worker node deployment takes 10 minutes! Then your job will run. If the node is already deployed the job starts immediately.

This is due to:

VM configuration
Tools dependencies installation
CernVM-FS configuration
SLURM installation and configuration

SLURM main commands¶

Please have a look here for a summary of SLURM main commands: https://www.rc.fas.harvard.edu/resources/documentation/convenient-slurm-commands/

Authentication¶

Currently, the authentication system relies on INDIGO-AAI.

To login into the portal, select the Sign in section on top-right:

Registration¶

It is needed to register to the portal at the first login. Register with your preferred username or using Google authentication.

Fill the registration form using a valid e-mail address:

and accept the usage policy to complete the registration:

A confirmation e-mail is the sent your e-mail address:

You don’t need to answer to this mail, just follow the instructions, going to the link in the e-mail.

Once confirmed, your request has to be approved by the site administrators. This usually does not require too much time.

Once your request is approved, you will be notified by mail and asked to insert your password.

Finally at the first login you have to allow the Laniakea portal to acquire your login information:

Validation¶

Warning

Validation package for automatic tools test is under development and it is not currently available.

Testing module for Galaxy workflows: https://github.com/phnmnl/wft4galaxy

Galaxyctl libraries¶

Galaxyctl is a python script collection for Galaxy management (first start, stop/start/restart/status). Moreover it is possible to manage, through specific script, LUKS volumes and Onedata spaces.

Galaxyctl requires superuser privileges.

Current version: 0.0.1

Script	Description
galaxyctl_libs	Python libraries for uWSGI socket and stats server management, LUKS volume and Onedata space management.
galaxyctl	Galaxy management script. It integrates Luksctl and Onedatactl commands.
luksctl	LUKS volume script management
onedatactl	Onedata spaces for user data and reference data management.

Galaxyctl_libs is composed by several modules.

Dependencies¶

Galaxyctl_libs depends on uWSGI for Galaxy management (i.e. currently no run.sh support). Moreover lsof is needed to check listening ports.

uwsgi

lsof

DetectGalaxyCommands¶

Parse galaxy Stop/Start/Restart/Status commands. Currently it supports supervisord or systemd/upstart

UwsgiSocket¶

Get uWSGI socket from galaxy.ini config file (e.g. 127.0.0.1:4001) and using lsof return uWSGI master PID.

master_pid, stderr, status = UwsgiSocket(fname='/home/galaxy/galaxy/config/galaxy.ini').get_uwsgi_master_pid()

UwsgiStatsServer¶

Read uWSGI stats server json. The stats server is the last software which uWSGI run during galaxy start procedure. When the stats server is ready, galaxy is ready to accept requests. Stats server address and port can be specified, but the class is able to read galaxy.ini file to recover stats informations. Reading Stats json the class is able to detect if uWSGI workers accept requests or not.

Inputs	Description
server	uWSGI stats server address, e.g. 127.0.0.1
port	uUWSG stats server port, e.g. 9191
timeout	Wait time, in seconds, for the Stats server start. If galaxy is starting, `300` seconds as timeout is ok, while if galaxy is already running `5` seconds are enough.
fname	Galaxy config file, e.g. /home/galaxy/galaxy/config/galaxy.ini

GetUwsgiStatsServer¶

To connect to running uWSGI stats server call:

stats = UwsgiStatsServer(timeout=300, fname='/home/galaxy/galaxy/config/galaxy.ini)
socket = stats.GetUwsgiStatsServer()

GetUwsgiStatsServer¶

To check if at least one uWSGI workers accept requests, call:

stats = UwsgiStatsServer(timeout=300, fname='/home/galaxy/galaxy/config/galaxy.ini)
status = stats.GetUwsgiStatsServer('/home/galaxy/galaxy/config/galaxy,ini')

GetBusyList¶

To get the list of busy uWSGI workers:

stats = UwsgiStatsServer(timeout=5, fname='/home/galaxy/galaxy/config/galaxy.ini)
busy_list = stats.GetBusyList()

LUKSCtl¶

Read LUKS ini file, usually stored on /etc/galaxy/luks-cryptdev.ini, for LUKS volume management. Open, Close and Status commands are managed through luksctl script.

OneDataCtl¶

Reads Onedata ini file, usually stored on /etc/galaxy/onedatactl.ini, for Onedata space management: both user data and reference data.

mount_space¶

To mount onedata space (userdata or refdata), call:

onedata = OneDataCtl('/etc/galaxy/onedatactl.ini', 'userdata')
onedata.mount_space()

umount_space¶

To umount onedata space, call:

onedata = OneDataCtl('/etc/galaxy/onedatactl.ini', 'userdata')
onedata.umount_space()

Galaxyctl: Galaxy management¶

Galaxyctl is a python script collection used for Galaxy management. In particular it exploits galaxyctl_libs to properly check uWSGI Stats sto correctly retrieve Galaxy and uWSGI workers status. Moreover the script allow to manage, using the same command, luks volume and onedata spaces, parsing luksctl and onedatctl commands.

Since the script parse supervisorctl or systemd commands, it needs to be run as superuser.

Moudule	Action	Description
galaxy	status	Show galaxy status
	stop	Stop Galaxy. `--force` check uwsgi master process. If it is still running, after galaxy stop, it is killed.
	start	Start Galaxy. `--force` force galaxy to start by restarting it. `--retry` option allow to specify number of tentative retart (default 5). `--timeout` allow to customize uWSGI stats server wait time. These options are used during galaxy instantiation and you should not use them on production.
	restart	Restart Galaxy. `--force` force galaxy to start by restarting it. `--retry` option allow to specify number of tentative retart (default 5). `--timeout` allow to customize uWSGI stats server wait time. These options are used during galaxy instantiation and you should not use them on production.
	startup	This method is used only to run galaxy for the first time and you shoud not use it in production. `--retry` option allow to specify number of tentative retart (default 5). `--timeout` allow to customize uWSGI stats server wait time.
luks	status	Show LUKS volume status
	open	LUKSOpen volume
	close	LUKSClose volume
onedata	status	Show onedata space status. `--userdata` allow to check user data space if mounte through oneclient. `--refdata` allow to check reference data onedata space if mounted through oneclient.
	mount	Mount onedata space using oneclient. Use `--userdata` or `--refdata` to mount user data and reference data volumes.
	umount	Umount onedata space using furermount.

Galaxyctl basic usage¶

The script requires superuser commands to be used. Its basic commands are:

Action	Command
Start Galaxy	sudo galaxyctl start galaxy
Stop Galaxy	sudo galaxyctl stop galaxy
Restart Galaxy	sudo galaxyctl restart galaxy
Check Galaxy Status	sudo galaxyctl status galaxy

Logging¶

Logs are stored in /var/log/galaxy/galaxyctl.log file.

Advanced options¶

stop¶

To stop galaxy:

sudo galaxyctl stop galaxy

The script check the uWSGI Stats server to retrieve workers PID and their status. If, after uWSGI stop, workers are still up and running, they are killed, allowing Galaxy to correctly start next time. The --force options allow to kill uwsgi master process if it is still alive after galaxy stop (in case of uwsgi FATAL error or ABNORMAL TERMINATION). Please check galaxy logs before run --force option.

start¶

To start Galaxy:

sudo galaxyctl start galaxy

Once Galaxy started, galaxyctl waits and check the uWSGI Stats server. Since it is the last software loaded, this ensure that Galaxy has correctly started. The script also check that at least 1 uWSGI worker has correctly started and it is accepting requests.

If no workers are available you have to restart Galaxy. Galaxyctl is able to automatically restart galaxy if the option --force is specified, restarting it until the workers are correctly loaded The number of retries is set, by default, to 5. It can be customized using --retry option, e.g. --retry 10. These options were not designed for production, but are used only during VMs instantiation phase to ensure Galaxy can correctly start.

restart¶

To restart Galaxy:

sudo galaxyctl restart galaxy

The options --force, --timeout and --retry are available for restart command too.

Galaxy first start¶

Galaxy takes longer to start the first time. Since the uWSGI stats server is the last software component started, the script waits it ensure that Galaxy has correctly started. Then uWSGI workers are checked to ensure Galaxy is acceptin requests. If not, uWSGI is restarted. Currently, before rise an error, the script try to restart galaxy 5 times, while the waiting time is set to 600 seconds. The command used in /usr/local/bin/galaxy-startup script, is

galaxyctl startup galaxy -c /home/galaxy/galaxy/galaxy.ini -t 600

LUKS module¶

By parsing luksctl script and using galaxyctl_libs, galaxyctl is able to manage LUKS volumes

Action	Command
Open LUKS volume	sudo galaxyctl open luks
Close LUKS volume	sudo galaxyctl close luks
Check LUKS volume	sudo galaxyctl status luks

In particular to unlock you LUKS volume:

sudo galaxyctl open luks

Then you will be asket to insert your LUKS passphrase. For instance:

(.venv) [galaxy@galaxy-indigo-test ~]$ sudo galaxyctl open luks
Enter passphrase for /dev/disk/by-uuid/42aaf979-6351-44e9-97ee-19e7f8c5e9f6:

Onedata module¶

By parsing onedatactl script and using galaxyctl_libs, galaxyctl is able to manage onedata user data and reference data spaces.

Data	Action	Command
User data	mount	sudo galaxyctl mount onedata –userdata
	umount	sudo galaxyctl umount onedata –userdata
	status	sudo galaxyctl status onedata –userdata
Refarence data	mount	sudo galaxyctl mount onedata –refdata
	umount	sudo galaxyctl umount onedata –userdata
	status	sudo galaxyctl status onedata –refdata

The options --userdata and --refdata are mutually exclusive.

Configuration files¶

Supervisord and systemd/upstart are supported to start/stop/restart/status Galaxy. The init system can be set using the variables init_system: two values are, currently, allowed: supervisord and init

init_system	Explanation
supervisord	Supervisord is current default, it is mandatory for docker container, since there’s no systemd on docker images.
init	CentOS 7 and Ubuntu 16.04 use systemd, while Ubuntu 14.04 is using upstart.

Through galaxyctl_libs.DetectGalaxyCommands method the script automatically retrieve the right command to be used and it is compatible with both CentOS 7 and Ubuntu (14.04 and 16.04).

If Supervisord is used to manage Galaxy (which is our default choice), configuration files have to be specified using the variable supervisord_config_file On CentOS:

supervisord_conf_file = '/etc/supervisord.conf'

while on Ubuntu:

supervisord_conf_file = '/etc/supervisor/supervisord.conf'

Galaxyctl needs galaxy.ini to retrieve uWSGI stats server information, through the variable:

galaxy_config_file = '/home/galaxy/galaxy/config/galaxy.ini'

For LUKS volume configuration, the script reads our custom luks-cryptdev.ini file (stored in /etc/galaxy/ and needs luksctl script path (usually stored in /usr/local/bin) to load methods

luks_config_file = '/etc/galaxy/luks-cryptdev.ini'
luksctl_path = '/usr/local/bin'

Finally, for onedata spaces management, onedatactl.ini file (stored in /etc/galaxy) and onedatactl path (usually /usr/local/bin) are needed:

onedatactl_config_file = '/etc/galaxy/onedatactl.ini'
onedatactl_path = '/usr/local/bin'

Luksctl: LUKS volumes management¶

Luksctl is a python script allowing to easily Open/Close and Check LUKS encrypted volumes, parsing dmsetup and cryptsetup commands.

The script requires superuser rights.

Action	Command
Open	sudo luksctl open
Close	sudo luksctl close
Status	sudo luksctl status

Dependencies¶

Since the script is going to parse cryptsetup and dmsetup commands, both are required

cryptsetup

Open LUKS volumes¶

To open LUKS volume, call: luksctl open, which will require your LUKS decrypt password:

$ sudo luksctl open
Enter passphrase for /dev/disk/by-uuid/9bc8b7c6-dc7e-4aac-9cd7-8b7258facc75:
Name:              ribqvkjj
State:             ACTIVE
Read Ahead:        8192
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 1
Number of targets: 1
UUID: CRYPT-LUKS1-9bc8b7c6dc7e4aac9cd78b7258facc75-ribqvkjj


Encrypted volume: [ OK ]

Close LUKS volumes¶

To Close LUKS volume, call luksctl close:

$ sudo luksctl close
Encrypted volume umount: [ OK ]

LUKS volumes status¶

To check if LUKS volume is Open or not call luksctl status

$ sudo luksctl status
Name:              ribqvkjj
State:             ACTIVE
Read Ahead:        8192
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 1
Number of targets: 1
UUID: CRYPT-LUKS1-9bc8b7c6dc7e4aac9cd78b7258facc75-ribqvkjj


Encrypted volume: [ OK ]

Onedatactl: Onedata spaces management¶

Onedatactl is a python script to mount Onedata spaces, for user data and reference data. This script parse oneclient options using a configuration file (onedatactl.ini) to retrieve the required onedata space information (e.g. oneprovider, token and mountpoint).

Options¶

Parsing all oneclient option is out of the purpose of this script. Therfore the script provides only few options to allow user to easily mount their data hosted on onedata

Option	Description
mount	Mount user data or reference data space.
umount	Umount user data or reference data space.
`-t` `--token`	Set access token.
`-H` `--provider`	Set onedata provider.
`-m` `--mountpoint`	Set mountpoint.
`-c` `config_file`	Load configuration file.
`--insecure`	Allow insecure mount.
`--nonempty`	Mount space on non empty space.
`--version`	Show galaxyctl_libs version.

The onedatactl.ini file¶

Onedatactl.ini file is located in /etc/galaxy/onedatactl.ini. The ini file has two secitons: [userdata] for user data management and refdata for reference data mangement.

mountpoint¶

By enabling this option onedata space will be used to store user data and it will be mounted on the specified directory, e.g. /export This argument is required.

mountpoint = /export

provider¶

Insert Onedata provider for user data, e.g. oneprovider2.cloud.ba.infn.it This argument is required.

provider = oneprovider2.cloud.ba.infn.it

token¶

Insert Onedata token for user data. This argument is required.

token = MDAxNWxvY2F00aW9uIG9uZXpvbmUKMDAzYmlkZW500aWZpZXIgeExqMi00xdFN3YVp1VWIxM1dFSzRoNEdkb2x3cXVwTnpSaGZONXJSN2tZUQowMDFhY2lkIHRpbWUgPCAxNTI1MzM00NzgyCjAwMmZzaWduYXR1cmUgIOzeMtypO75nZvPJdAocInNbgH9zvJi6ifgXDrFVCr00K

insecure¶

Allow insecure connection. This is an optional argument.

insecure = False

nonempty¶

Allow to mount space on non empty directory. This is an optional argument

nonempty = False

Onedatactl options¶

The script allow to mount two different volumes.

mount¶

For user data, call:

sudo onedatactl mount userdata

For reference data, call:

sudo onedatactl mount refdata

umount¶

For user data, call:

sudo onedatactl umount userdata

For reference data, call:

sudo onedatactl umount refdata

status¶

For user data, call:

sudo onedatactl status userdata

For reference data, call:

sudo onedatactl status refdata

Oneclient¶

Onedatactl parse oneclient command using the information stored in /etc/galaxy/onedatactl.ini file. Therefore oneclient can be used to perform the same actions.

The basic command line syntax to mount spaces using a specific Oneprovider is:

oneclient -H <PROVIDER_HOSTNAME> -t <ACCESS_TOKEN> <MOUNT_POINT>

In order to unmount your spaces, type:

oneclient -u MOUNT_POINT

The complete Oneclient documentation is located here: https://onedata.org/docs/doc/using_onedata/oneclient.html

Troubleshooting¶

If you are connecting to a provider service which does not have a globally trusted certificate, you will have to use --insecure option.

Frequently Asked Questions¶

Laniakea FAQs.

Recover Galaxy after Virtual Machine reboot¶

How to correctly restart Galaxy after a reboot of the Virtual Machine ?

After the boot procedure of your VM at least these services should already be up and running:

PostgreSQL
Proftpd
NGINX

Trying to connect to your Galaxy instance IP with a web browser you should see:

Step 1: Unlock encrypted storage¶

If your instance is not mounted on encrypted storage please skip this and go to Step 2: One-command procedure

To unlock the volume connect with SSH and type the command sudo /usr/local/bin/luksctl open followed by your passphrase.

$ sudo /usr/local/bin/luksctl open
Enter passphrase for /dev/disk/by-uuid/4ba96890-f914-46aa-b7a6-7e71f0846f43:

Name:              jzwoejuw
State:             ACTIVE
Read Ahead:        8192
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      252, 0
Number of targets: 1
UUID: CRYPT-LUKS1-4ba96890f91446aab7a67e71f0846f43-jzwoejuw

Encrypted volume: [ OK ]

Step 2: One-command procedure¶

To start Galaxy you can just copy and paste this:

$ sudo su -c "wget -O - https://raw.githubusercontent.com/Laniakea-elixir-it/Scripts/master/galaxy/recover.sh | bash" root

Probing /cvmfs/elixir-italy.galaxy.refdata... OK
Contacting Galaxy (wait for 10 seconds)...
Galaxy server on-line: [ OK ]

Warning

This may require few minutes, but should safely restart your Galaxy.

Advanced Step-by-step procedure¶

Warning

Please follow this instructions only if you know what you are doing!

Mount reference data¶

After encrypion storage unlock (see section Step 1: Unlock encrypted storage), if you have it, you need now to mount your reference datasets:

$ sudo systemctl restart autofs

$ sudo cvmfs_config killall

Terminating cvmfs_config processes... OK
Terminating cvmfs2 processes... OK
Unmounting stale mount points... OK
Cleaning up run-time variable data... OK
Reloading autofs... OK

$ sudo cvmfs_config probe

Probing /cvmfs/elixir-italy.galaxy.refdata... OK

Start Galaxy¶

Finally, to correctly start all Galaxy services run: sudo /usr/local/bin/galaxy-startup

$ sudo /usr/local/bin/galaxy-startup

Loading Galaxy environment
Exporting environment variables
Check if galaxy is up and running, to avoid unuseful restart
status: 502
Galaxy unreachable
Check is supervisord is running
Starting the Galaxy production environment
/usr/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated.  Call .resolve and .require separately.
return pkg_resources.EntryPoint.parse("x="+spec).load(False)

Galaxy start: [ OK ]

Galaxycloud Ansible Roles¶

Ansible automates Galaxy (postgresql, NGINX, uWSGI, proftpd) installation and configuration using YAML syntax.

These roles make extensive use of Ansible Modules, which are the ones that do the actual work in ansible, they are what gets executed in each playbook task. Furthermore, a python scripts collection for galaxy advanced configuration is used (run by ansible).

All roles can be easily installed through ansible-galaxy, for instance:

ansible-galaxy install indigo-dc.galaxycloud

Ansible roles documentation¶

Galaxycloud¶

Install Galaxy Production environment. This role has been specifically developed to be used for the ELIXIR-IIB use case in the INDIGO-DataCloud project.

Requirements¶

This ansible role supports CentOS 7, Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial

Note

Minimum ansible version: 2.1.2.0

Role Variables¶

Path¶

galaxy_instance_description: set Galaxy brand

galaxy_user: set linux user to launch the Galaxy portal (default: galaxy).

GALAXY_UID: set user UID (default: 4001).

galaxy_FS_path: path to install Galaxy (default: /home/galaxy).

galaxy_directory: Galaxy directory (usually galaxy or galaxy-dist, default galaxy).

galaxy_install_path: Galaxy installation directory (default: /home/galaxy/galaxy).

galaxy_config_path: Galaxy config pat location.

galaxy_config_file: Galaxy primary configuration file.

galaxy_venv_path: Galaxy virtual environment directory (usually located to <galaxy_install_path>/.venv).

galaxy_custom_config_path: Galaxy custom configuration files path (default: /etc/galaxy).

galaxy_custom_script_path: Galaxy custom script path (defautl: /usr/local/bin).

galaxy_log_path: log file directory (default: /var/log/galaxy).

galaxy_instance_url: instance url (default: http://<ipv4_address>/galaxy/).

galaxy_instance_key_pub: instance ssh public key to configure <galaxy_user> access.

galaxy_lrms: enable Galaxy virtual elastic cluster support. Currently supported local and slurm (default: local, possible values: local, slurm).

main options¶

GALAXY_VERSION: set Galaxy version (e.g. master, release_17.01, release_17.05…).

create_galaxy_admin: if true the administrator user will be created (default: true).

GALAXY_ADMIN_USERNAME: Galaxy administrator username.

GALAXY_ADMIN_PASSWORD: Galaxy administrator password.

GALAXY_ADMIN_API_KEY: Galaxy administrator API_KEY. https://wiki.galaxyproject.org/Admin/API. Please note that this key acts as an alternate means to access your account, and should be treated with the same care as your login password. To be changed by the administrator.(default value: GALAXY_ADMIN_API_KEY)

GALAXY_ADMIN_EMAIL: Galaxy administrator e-mail address

Galaxy configuration¶

export_dir: Galaxy userdata are stored here (defatult: /export).

tool_deps_path: change tool dependency directory (default: {{ export_dir }}/tool_deps)

use_conda: enable Conda (default: true).

job_work_dir: change job_working_dir path. Due to a current limitation in conda, the total length of the conda_prefix and the job_working_directory path should be less than 50 characters! (default: {{ export_dir }}/job_work_dir).

conda_prefix: change conda prefix directory (default: {{ export_dir }}/_conda).

conda_channels: change conda channels (default: iuc,bioconda,r,defaults,conda-forge).

update_ucsc: update UCSC genome database (default: ``true). A monthly cron job is added to keep update ucsc genome db.

fast_update: force database update by copying cached files (default: true).

use_pbkdf2: enable pbkdf2 cryptograpy (default: true).

Postgres database details¶

postgresql_version: set postgres version to be installed (current default: 9.6).

galaxy_db_dir: change galaxy database directory to store jobs results (default: {{export_dir}}/galaxy/database).

galaxy_db_port: set postgres port (default: 5432).

galaxy_db_passwd: set database password. By default it is generated a random password 20 characters long.

set_pgsql_random_password: if set to false the role takes the password specified through galaxy_db_passwd variable (default: true).

NGINX¶

nginx_upload_store_path: set nginx upload dataset directory (default: {{galaxy_db_dir}}/tmp/nginx_upload_store).

https mode¶

The Galaxy portal runs through an nginx http proxy by default. The following variables enable you to set nginx in https mode:

nginx_https: true
ssl_cert: /etc/certs/cert.pem
ssl_key: /etc/certs/key.pem
ssl_dhparam: /etc/certs/dhparam.pem

If nginx_https is set to true, the other ssl variables are required. You can either request a signed trusted certificate or generated self-signed certificate. An ansible role to generate self-signed certificate can be found in https://galaxy.ansible.com/LIP-Computing/ssl-certs/

PROFTPD¶

proftpd_welcome: set proftpd welcome message (default: galaxy ftp server).

proftpd_conf_path: set proftpd configuration file path.

proftpd_db_user: set proftpd database user (default: galaxyftp).

proftpd_db_passwd: set postgresql database password. By default it is generated a random password 20 characters long.

proftpd_files_path: set proftpd upload directory (default: {{galaxy_db_dir}}/ftp).

proftpd_ftp_port: set proftpd port (default: 21).

proftpd_passive_port_low: set passive port range minimum (default: 30000).

proftpd_passive_port_high: set passive port reng maximum (default:40000).

set_proftpd_random_password: if set to false the role takes the password specified through proftpd_db_passwd variable (default: true).

Init system¶

Currently this role support supervisord and systemd/upstart to start Galaxy services. init_type: if set to supervisord, it use to manage Galaxy. If set to init systemd/upstart is used to start Galaxy.

It is possible to exploit supervisord to manage postegreSQL, NGINX and proftpd setting to true the following variables. To run this role on docker container you have to set them to true. supervisor_manage_postgres: enable supervisord postgresql management (default: false).

supervisor_manage_nginx: enable supervisord nginx management (default: false).

supervisor_manage_proftpd: enable supervisord proftpd management (default: false).

Advanced storage configuration¶

enable_storage_advanced_options: this option, false by the default, has to be set to true only if you run the ansible role indigo-dc.galaxycloud-os, for advanced path configuration, onedata and filesystem encryption support. More details here: Galaxycloud-os (default: false).

Example Playbook¶

Including an example of how to use your role (for instance, with variables passed in as parameters) is always nice for users too:

- hosts: servers
  roles:
     - role: indigo-dc.galaxycloud
       GALAXY_ADMIN_EMAIL: "admin@elixir-italy.org"
       GALAXY_ADMIN_USERNAME: "admin"
       GALAXY_VERSION: "release_17.05"
       galaxy_instance_key_pub: "your_public_key"
       galaxy_instance_description: "INDIGO-CNR test"

Install Galaxy setting postgresql passwords:

- hosts: servers
  roles:
     - role: indigo-dc.galaxycloud
       GALAXY_ADMIN_EMAIL: "admin@elixir-italy.org"
       GALAXY_ADMIN_USERNAME: "admin"
       GALAXY_VERSION: "release_17.05"
       galaxy_instance_key_pub: "your_public_key"
       galaxy_instance_description: "INDIGO-CNR test"
       set_pgsql_random_password: false
       galaxy_db_passwd: 'galaxy'
       set_proftpd_random_password: false
       proftpd_db_passwd: 'galaxy'

Setup Galaxy Docker container. The role, using ansible, automatically recognize the virtual platform (virtual machine or Docker contanier).

- hosts: servers
  roles:
     - role: indigo-dc.galaxycloud
       GALAXY_ADMIN_EMAIL: "admin@elixir-italy.org"
       GALAXY_ADMIN_USERNAME: "admin"
       GALAXY_VERSION: "release_17.05"
       galaxy_instance_key_pub: "your_public_key"
       galaxy_instance_description: "INDIGO-CNR test"
       supervisor_manage_postgres: "True"
       supervisor_manage_nginx: "True"
       supervisor_manage_proftpd: "True"

License¶

Apache Licence v2

References¶

Galaxy: https://galaxyproject.org/

Apache licence: http://www.apache.org/licenses/LICENSE-2.0

Galaxycloud-os¶

This role provides advanced storage options for Galaxy instances.

Warning

Run indigo-dc.galaxycloud-os before indigo-dc.galaxycloud, setting the variable enable_storage_advanced_options to true.

It is possible to select three different storage options using the os_storage ansible role variable.

Storage provider	Description
Iaas	IaaS block storage volume is attached to the instance and Galaxy is configured.
onedata	Onedata space is mounte through oneclient and Galaxy is configured.
encryption	IaaS block storage volume is encrypted with aes-xts-plain64 algorithm using LUKS.

Path configuration for Galaxy is then correctly set, depending on the storage solution selected, replacing the indigo-dc.galaxycloud path recipe (with the enable_storage_advanced_options set to true).

The role exploits the galaxyctl_libs (see Galaxyctl libraries) for LUKS and onedata volumes management .

LUKS encryption¶

For a detailed description of LUKS encryption used and scripts, see section Storage encryption.

Dependencies¶

For LUKS encryption the ansible role install cryptsetup.

For onedata reference data provider, the role depends on indigo-dc.oneclient role, to install oneclient:

- hosts: servers
  roles:
    - role: indigo-dc.oneclient
      when: os_storage == 'onedata'

Variables¶

The Galaxy path variables are the same of indigo-dc.galaxycloud.