NEF-Pipelines

Tools for Manipulating NEF [NMR Exchange Format] Files and Foreign File Access

Please note NEF-Pipelines continues to be developed so if you do find problems put in an issue on the issue tracker. Even better if you can help improve the software do get in touch, there is much to do!

NEF-Pipelines is a set of command line (currently... there maybe a gui later!) tools for manipulating [NEF])(https://github.com/NMRExchangeFormat/NEF) or NMR Exchange Format files which can be used to move NMR meta data [peaks, shifts, sequences etc] between NMR data processing programs.The programs provide basic tools for manipulating nef files:

• molecular chains: listing, renaming and cloning molecular chains
• save-frames [tables]: deleting, inserting, listing and pretty printing (tabulate)
• headers creation / updating NEF headers with correct UUIDs and history
• streaming: NEF files into a pipeline
• testing: self testing of NEF pipelines
• accessing online data: chemical shift predictions
• fitting data: basic relaxation data, j-modulated RDC experiments [in development]

It also provides tools for importing and exporting non NEF files from the following programs [transcoding / translators]

R = Read / import to NEF, W = write / export from NEF, 🐍= supported in python code, α = alpha quality code [few tests]

xplor dihedrals R distances R sequence R rdcs W	csv peaks R 🐍 rdcs R 🐍	deep peaks R α 🐍	nmrpipe peaks R 🐍 sequence R 🐍 shifts R 🐍	nmrview peaks RW 🐍 sequence R shifts RW 🐍 sequences W 🐍
fasta sequence RW 🐍	echidna peaks R 🐍	sparky shifts R 🐍 peaks RW 🐍 sequence R 🐍	mars fixed W 🐍 fragments W 🐍 input W 🐍 sequence RW 🐍 shifts RW 🐍 peaks R 🐍	modelfree data W α 🐍
pales rdcs RW 🐍 template W	rcsb sequence R	rpf shifts W α 🐍	shifty shifts W 🐍	shiftx2 shifts R α 🐍
nmrstar project R α rdcs R α 🐍 sequence R α 🐍 shifts R α 🐍	talos order-parameters R 🐍 restraints R α 🐍 secondary-structure R 🐍 sequence R 🐍 shifts W α 🐍	xcamshift shifts W 🐍	xeasy peaks R 🐍 sequence R 🐍 shifts R 🐍

Installation

The easiest way to install NEF-Pipelines is using the tool uv provided by astral. You can do this either using the script installation_scripts/install.sh from the NEF-Pipelines distribution [on macos or a linux] as shown below or alternatively using uv manually [macos, windows and linux] as shown further down

macOS or Linux Using the Script install.sh

```bash

curl -sL https://raw.githubusercontent.com/varioustoxins/NEF-Pipelines/refs/heads/master/install_scripts/install.sh | sh ```

Manual installation on macOS or Linux using uv

bash curl -LsSf https://astral.sh/uv/install.sh | sh uv tool install nef-pipelines --with rich --with streamfitter --python 3.11

note: currently it is recommended to install with a python version <= 3.11 on macOS as there is a bug in installing a dependency call pydantic on some version of the OS

Manual Installation on Windows using uv

```powershell

note: changing the execution policy is required for running a script from the internet.

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" uv tool install nef-pipelines --with rich ```

note: not all of NEF-Pipelines is working on Windows currently, we are working on it!

after the installation has completed you can type

bash nef

Checking NEF-Pipelines is Installed

type bash nef and should see output that starts

``` Usage: nef [OPTIONS] COMMAND [ARGS]...  ✔ ╱ took 14s  ╱ nef_pipelines  ╱ at 23:29:36 

Options: ... ```

Older installations methods

Some older installation methods using a script in ccpn assign or pip are deprocated but still provided for completenes

Updating NEF-Pipelines

You can use the same tools you used to in stall NEF-Pipelines to upgrade it

Updating NEF-Pipelines on macOS or Linux Using the Script install.sh

running the following script again bash curl -sL https://raw.githubusercontent.com/varioustoxins/NEF-Pipelines/refs/heads/master/install_scripts/install.sh | sh will update NEF-Pipelines to the latest version

Updating NEF-Pipelines on macOS, Windows or Linux Manually Using UV

again now you have UV installed upgrading NEF-Piplines manually should be easy bash uv tool upgrade nef-pipelines --python 3.11

Using NEF-Pipelines on the command line

The Basic nef Command

The entry point to NEF-Pipelines is the command nef used on its own as seen under installation it will list a set of top level commands and gives there descriptions.

```bash nef Usage: nef [OPTIONS] COMMAND [ARGS]...

...

Commands: chains - carry out operations on chains csv - read [rdcs] deep - read deep [peaks] echidna - read echidna data [peaks] entry - carry out operations on the nef file entry fasta - read and write fasta sequences fit - carry out fitting operations [alpha] frames - carry out operations on frames in nef files globals - add global options to the pipeline [use save as your last... header - add a header to the stream help - help on the nef pipelines tools and their usage loops - carry out operations on loops in nef frames mars - read and write mars [shifts and sequences] ... `` Some of these commands such asframesandentitydeal with components of a NEF file. Then there are a series of top levels that deal with common SaveFrames related to NMR data such aschainsandshiftsand finally there are many commands named after file formats and programs such astalos,xplornmrpipeandnmrview` which provide entry points to the import and export of data.

Invoking and Finding Sub-commands

NEF-Piplines uses hierarchical commands. So foir example the the toip level command mars which deals with the import and export of data from the mars auto assignment program has two sub-commands import ane export which not suprising deal with the import and export of data for the program to and from NEF. Then below these commands are further sub commands deaing with shifts, sequences etc. An easy way to see the sub commands for a format is to use the nef help commands tool which provides a tree display of the available commands. Typing nef help commands on its own produces output for all commands which can be overwhelming so lets filter it down to just talos commands by adding a filter. To do this we type nef help commands mars which gives the following output

```bash nef └── mars ├── export │ ├── fixed [P] │ ├── fragments [P] │ ├── input [P] │ ├── sequence │ └── shifts [P] └── import ├── peaks [P] ├── sequence [P] └── shifts [P]

key: [X] has a python function [P]ipe / [C]md [α] alpha feature `` here we can see that we can import peaks, shifts and sequencdes from mars and that we export fixed assignments, fragments, inputs [an input file], sequences and shifts to mars. Each individual file sub-command also has help text and a series of options. So to see the options for exproting shift we can typenef mars export shifts --help` which will show

```bash nef mars export shifts --help Usage: nef mars export shifts [OPTIONS] [SHIFTFRAMESELECTORS]...

• write a mars chemical shift file

Arguments: [SHIFTFRAMESELECTORS]... selector for the shift restraint frames to use, can be called multiple times and include wild cards

Options: -i, --input |PIPE| input to read NEF data from [- is stdin] [default: -] -c, --chain chain to export [# and @ and the 1st chain code in the project if the is only one] -o, --out file name to output to [default _shifts.tab] for stdout use - -f, --force force overwrite of output file if it exists and isn't empty --help Show this message and exit. `` However, mostly you don't need to worry about many of the optiosn as the defaults are well chosen. It should be noted also that many commands also take arguments in the case ofmars export shifts` the default is the default chemical shift frame (again a well chosen default).

Common Options of Sub-commands

many commands also have common options, the most well used ones are

| option | description | |---------------------------|------------------------------------------------------------------------------------------------| | -i / --input | the path to a file to read NEF data from which becomes the stream running through the pipeline | | -h / --help | help for the particular sub command | | | -o / --out | a filename or a template file name to write output to [usually native files not NEF]
| | -c / --chain | a chain or list of chains to output, multiple invocations add more chains and the argument can also be comma separated list [don't add any spaces!]| | -f / --force | force overwriting files if newly generated files would overwite old ones| | -v / --verbose | provide verbose information on the STDERR stream, in some cases repeating the option gives more output, e.g -vv gives more output the -v'|

Note 1. It should be noted that many options that take multiple values can be repeated to add more values or can or can take comma separated lists of values [no spaces] so

bash command --chain A --chain B ``` selects chains A and B as does

bash command --chains A,B Note 2. Many arguments take wild cards, so when selecting frames you can use +'s and * when choosing the frame name or category to select multiple frames. Furthermore all frame names are typically treated as being surrounded by *s when it is reasonable to make selecting frames easier

Note 3. When selecting frames there is usually an explicit option to allow differentiation between frame names or categories. However, by default it is off and searches for frames are based on both names and categories the same time

Using Individual NEF-Pipelines Commands

Individual nef pipeline components can be used standalone with NEF files. To read in a nef fileo or fiel for conversion into a command when using it standalone you can use the --in option read a foreign file for translators or read a NEF file for commands that manipulate NEF files. So for example to lists the frames in NEF frame you can type

bash nef frames list --in test_data/pales_test_1.nef will list all the Saveframes in the file testdata/palestest_1.nef and the entry name of the nef file as follows

```bash entry pales_test

nefmolecularsystem nefrdcrestraintlisttest_1 ```

as a short cut the following command also work for nef frames list

bash nef frames list test_data/pales_test_1.nef

Using NEF-Pipelin commands in pipelines

The real power in NEF-Pipelines comes from combining pipeline commands together or with command line tools. For example the following command

```bash nef header \ | nef fasta import sequence tailin1.fasta \ | src/nef nmrview import shifts tailin1.out \ | src/nef nmrview import peaks tailin1.out \

tailin1.nef ```

will create a valid NEF header followed by the tailin1 sequence as a molecular system and then import shifts and peaks for talin1 from nmrview before writing a new NEF file tailin1.nef

Command Line Completion

As discussed above the commands provided by NEF-Pipelines are hierarchical in nature. All commands are accessed from the root nef command but there many are sub commands so for example to import a sequence from a pdb file you would type nef rcsb import sequence followed by the name of a PDB / MMCIF file and relevant options. This can look long winded but NEF-Pipelines support command completion. This is is installed for your shell by typing nef --install completion <SHELL-NAME> and the restarting your shell, so fro example for the bash shell

bash nef --install-completion bash

The after restarting you shell, typing a double tab will list all available sub commands.

For example bash nef<tab><tab> shows

chains fasta frames header mars nmrpipe nmrview pales pdb stream test

and

bash nef nmrview import<tab<tab>

shows

peaks sequence shifts

note: currently I find this works better with the bash shell rather than the zsh but your mileage may vary...

Using NEF-Pipelines as Python Library

NEF-Pipelines maybe used as library of routines that are acessible in python using

NEF-Piplines workflows

in development...

More Information on NEF

The origional NEF paper is not a good description of the format more an outline of ideals and needs. The CCPN website has a good guide to nefand development takes place on github via the NEF consortium repository which includes an annotated NEF example and a dictionary defining the format. NEF is a STAR format which means it has similarities to NMRStar files used by the BMRB and PDBx / mmCIF used by the RCSB / PDB because its uses the same underling format. However, its is not directly interchangeable with these file formats as it has a different syntax and intention / underlying use easy and accurate NMR data interchange] as opposed to archiving for example.