Skip to content

Further methods for Importing Data§

XNAT is primarily designed around medical imaging data (typically DICOM) but can store any types of files as well as custom data types. The XNAT developers provide easy to use methods and tools for importing both DICOM and other types of data to an XNAT server. Additionally, XNAT provides a comprehensive REST application programming interface (API) for programmatically exploring the data on an XNAT server as well as import and export of data.

This section is primarily aimed at developers and will briefly describe libraries and methods of importing data programmatically using the XNAT REST API. It will also briefly describe user interface tools provided by the XNAT developers for importing DICOM and other types of data - if this is of more interest to you, see our documentation here.

Common methods for importing data to XNAT§

User interfaces§

The XNAT Desktop client§

The XNAT desktop client is a lightweight, easy to install, application with a user interface which allows a user to log in to an XNAT server, explore the DICOM data in any projects they have access to and to upload additional DICOM data to that project (creating any new subjects or experiments) as needed (as well as providing a means for downloading data). If you are only interested in using the XNAT desktop client, see our brief user guide.

The interface does not at present provide a means for bulk uploading data but all data for a session/experiment can be added at the same time. Uploads can be queued, monitored, paused, resumed and cancelled in the user interface. If you have data ready to upload (i.e. not on a hospital PACS), and are able to install software on the machine you want to upload from, the XNAT desktop client is the preferred method for uploading.

If your data is not anonymised,the XNAT desktop client will be able to perform this anonymisation before the data is uploaded to XNAT (the UCL XNAT servers cannot be used to store identifiable information). An anonymisation script based on a custom profile can be stored in your project on the XNAT server. When using the desktop client this anonymisation script will be downloaded and used by the XNAT desktop client to anonymise the data before upload. If you need information or guidance about anonymisation please talk to member of the MIRSG - it is your responsibility to ensure that your data is either anonymised prior to uploading, whether this is done in a separate step or using a script on the XNAT server.

The XNAT web interface§

The XNAT web interface can be used to upload both DICOM and non-DICOM to your project on XNAT. A more comprehensive description can be found here or in the XNAT documentation. Briefly, image sessions can be compressed using ZIP and uploaded using the compressed uploader while non-DICOM data can be added using the XNAT File Manager.

Using the XNAT REST API§

XNAT has a comprehensive REST API which can be used to configure and explore the server as well as uploading and downloading files. Site administrators can use the Swagger interface to explore the API functionality. Non-admin developers should look at the XNAT REST API documentation to find out what functionality is provided.

Command line§

Command line tools such as cURL can be used to make calls to the XNAT REST API.

Some examples of REST API calls are:

Get a list of subjects

curl -u XNAT_USERNAME -X GET "SERVER/data/subjects"

Get a list of resource files for a specific subject

curl -u XNAT_USERNAME -X GET "SERVER/data/projects/{project-label}/subjects/{subject-id | {subject-label}}/files"

Get the contents of a resource file and save to disk

curl -u XNAT_USERNAME -X GET "SERVER/data/projects/{project-label}/subjects/{subject-id | {subject-label}}/resources/{resource-label}/files/{filename}" -o "{output-filename}"

You will be prompted for your XNAT password when making these calls. If you don't want to be prompted you can pass the password as an argument to the cURL call:

curl -u XNAT_USERNAME:XNAT_PASSWORD {rest of the call}

The XNAT Data Client§

The Xnat Data Client (XDC) is built specifically to facilitate user interaction with the REST API from the command prompt. It is similar to cURL, and can even be used to make requests of non-XNAT HTTP services, but is XNAT "aware". For example, suppose you want to retrieve all of the scan files that compose a complete DICOM session. This will generally consist of quite a few files. With cURL, you would need to make a call to retrieve each file. Using XDC, you can make a single call to retrieve the session data and then have XDC follow the URLs for all of the constituent scan files and automatically download them. The number of HTTP calls is still the same, but XDC automates the process for you.

The following example shows how to create a subject, a scan and the final upload of files to the scan:

//if you configure the .xnatPass file in your home directory, you can skip adding the host, username, & password to the environment

// add server URL, XNAT username and password to the environment variables
setenv rest_params "-host SERVER -u USERNAME -p PASSWORD "

//create subject
XNATRestClient $rest_params -m PUT -remote "/data/projects/{project-label}/subjects/{subject-label}"

//create session (MR Session in this example)
XNATRestClient $rest_params -m PUT -remote "/data/projects/{project-label}/subjects/{subject-label}/experiments/{session-label}?xnat:mrSessionData/date={dd/mm/yy}"

//create SCAN in session (T1 MR data in this example)
XNATRestClient $rest_params -m PUT -remote "/data/projects/{project-label}/subjects/{subject-label}/experiments/{session-label}/scans/{scan-label}?xsiType=xnat:mrScanData&xnat:mrScanData/type=T1"


//upload SCAN files (T1_RAW in this example)...
XNATRestClient $rest_params -m PUT -remote "/data/projects/{project-label}/subjects/{subject-label}/experiments/{session-label}/scans/{scan-label}/resources/DICOM/resources/DICOM?format=DICOM&content=T1_RAW"

Python§

There are currently two Python libraries available which wrap the XNAT REST API.If you'd prefer not to use these libraries the Python Requests library can be used directly. Using a programmatic interface to the REST API provides consistency and repeatablitiy, as well as the convenience of being able to easily batch operations. A programmatic interface also allows integration with analysis pipelines.

pyXNAT§

pyXNAT provides an easy to use programmatic interface to the XNAT REST API using requests and python-lxml.

pyXNAT is easily installed (remember to use a virtual environment):

pip install pyxnat

Connect to a XNAT server (https://ucl-open-xnat.cs.ucl.ac.uk in this example):

>>> from pyxnat import Interface
>>> interface = Interface(server='https://ucl-open-xnat.cs.ucl.ac.uk',
                          user='login',
                          password='pass')

If you'd prefer not to have your password visible as plain text in code, you can access your credentials (in an access-protected location) from a config file:

>>> central = Interface(config='central.cfg')

The config file can be saved as follows:

>>> central.save_config('central.cfg')

When using the save_config method you'll first need to create an Interface instance omitting the user and password keywords - this will prompt you to enter the credentials:

>>> central = Interface(server="https://ucl-open-xnat.cs.ucl.ac.uk")
    User:my_login
    Password:

You can also use XNAT's config file format:

Create a file called ".xnatPass" in your home directory, formatted as follows:

+loginone@https://ucl-open-xnat.cs.ucl.ac.uk=password
-logintwo@https://ucl-open-xnat.cs.ucl.ac.uk=password
-logintwo@http://localhost=password

After your credentials are safely stored, a connection to the server can be created as follows:

>>> central = Interface()

Exploring the structure of data on the server can be done using the structure() method of the inspect sub-interface. Data selection and filtering is performed using the select sub-interface:

>>> central.select.projects().get()
[..., 'CENTRAL_OASIS_CS', 'CENTRAL_OASIS_LONG', ...]

Or specifying a path:

>>> central.select('/projects').get()
[..., 'CENTRAL_OASIS_CS', 'CENTRAL_OASIS_LONG', ...]

A filtered request can be made as follows:

>>> central.select.projects('*OASIS_CS*').get()
>>> central.select('/projects/*OASIS_CS*').get()
['CENTRAL_OASIS_CS']

>>> central.select.project('IMAGEN').subjects('*55*42*').get()
>>> central.select('/projects/IMAGEN/subjects/*55*42*').get()
['IMAGEN_000055203542', 'IMAGEN_000055982442', 'IMAGEN_000097555742']

Objects can be created or deleted on the server as follows:

>>> project = central.select.project('my_project')
>>> project.exists()
False
>>> project.insert()
>>> project.exists()
True
>>> subject = project.subject('first_subject')
>>> subject.insert()
>>> subject.delete()
>>> subject.exists()
False

Files can be uploaded to the server using the insert() method:

>>> project.resource('NIFTI').file('T1.nii').insert('/tmp/image.nii')
>>> project.resource('NIFTI').file('image.nii').insert(
...         '/tmp/image.nii',
...         content='T1',
...         format='NIFTI'
...         tags='image test')

XNATpy§

This is the newer of the two Python libraries and the main point of difference to pyXNAT is that it exposes XNAT objects/functions as python objects/functions. This makes using XNATpy more 'pythonic'.

Installation (again inside a virtual environment):

pip install xnat

Connections to the server can be made as follows (again using https://ucl-open-xnat.cs.ucl.ac.uk as an example):

>>> import xnat
>>> session = xnat.connect('https://central.xnat.org')

Sessions should always be closed:

>>> session.disconnect()

The xnat.connect() method has a context manager, which means that a self-closing session can be constructed:

>>> with xnat.connect('http://my.xnat.server') as session:
...     print(session.projects)

Credentials are read from a ".netrc" file stored in your home directory (or some other access-protected location). Such a file can be saved as follows:

echo "machine images.xnat.org
>     login admin
>     password admin" > ~/.netrc
chmod 600 ~/.netrc

An exception will be thrown if a ".netrc" doesn't exist or if the credentials are not stored correctly.

The server can easily be explored as follows:

>>> import xnat
>>> session = xnat.connect('http://images.xnat.org', user='admin', password='admin')
>>> session.projects
<XNATListing (sandbox, sandbox project): <ProjectData sandbox project (sandbox)>>

The XNATListing is a special type of mapping in which you can access elements by a primary key (usually the ID or Accession #) and a secondary key (e.g. the label for a subject or experiment). Selection can be performed the same as a Python dict:

>>> sandbox_project = session.projects["sandbox"]
>>> sandbox_project.subjects
<XNATListing (XNAT_S00001, test001): <SubjectData test001 (XNAT_S00001)>>

You can browse the following levels on the XNAT server: projects, subjects, experiments, scans, resources, files. Also under experiments you have assessors which again can contain resources and files. This all following the same structure as XNAT. Note that traversing all subjects/experiments on a server can be a time consuming process - for efficiency go down through the project level.

XNATListing objects behave as a Python dict and can iterated as follows:

>>> sandbox_project.subjects.keys()
[u'XNAT_S00001']
>>> sandbox_project.subjects.values()
[<SubjectData test001 (XNAT_S00001)>]
>>> len(sandbox_project.subjects)
1
>>> for subject in sandbox_project.subjects.values():
...     print(subject.label)
test001

Objects are easily created on the server as follows:

>>> import xnat
>>> connection = xnat.connect('https://xnat.example.com')
>>> project = connection.projects['myproject']
>>> subject = connection.classes.SubjectData(parent=project, label='new_subject_label')
>>> subject
<SubjectData new_subject_label>

Sessions can also be created using connection.classes. These classes are generated by XNATpy based on the datatypes available on the server. This includes any custom datatypes that have be added to the server using custom plugins. To see the available session datatypes run:

>>> dir(connection.classes)

Data can be uploaded to XNAT using the compressed uploader mechanism:

>>> session.services.import_('/path/to/archive.zip', project='sandbox', subject='test002')

XNAT will automatically attempt to route the data in archive.zip to the correct location.

There is also a import_dir convenience method which performs the compression step (compression can occur in-memory or on-disk) before uploading:

>>> session.services.import_dir('/path/to/data', project='sandbox', subject='test002')

Data can be sent to the prearchive when uploading by passing destination='/prearchive' to either import method. Users with administrator access can then review the data before it is finally archived.

Data in the prearchive can be moved to the archive programmatically as follows:

>>> prearchive_session = session.prearchive.sessions()[0]
>>> experiment = prearchive_session.archive(subject='ANONYMIZ3', experiment='ANONYMIZ3')
>>> print(experiment)
<MrSessionData ANONYMIZ3 (demo_E00092)>

Non-DICOM resources have a convenience upload_dir method:

Create a resource and upload files:

>>> resource = connection.classes.ResourceCatalog(parent=subject, label="RESOURCE_LABEL")
>>> resource.upload_dir('path/to/files')