Added VolumeServer

docs/volume-server/README.md

+What is VolumeServer
+=====================
+
+VolumeServer is a service for accessing subsets of volumetric density data. It automatically downsamples the data depending on the volume of the requested region to reduce the bandwidth requirements and provide near-instant access to even the largest data sets.
+
+It uses the text based CIF and BinaryCIF formats to deliver the data to the client. 
+
+For quick info about the benefits of using the server, check out the [examples](examples.md).
+
+Installing the Server 
+=====================
+
+- Install [Node.js](https://nodejs.org/en/) (tested on Node 6.* and 7.*; x64 version is strongly preferred).
+- Get the code.
+- Prepare the data.
+- Run the server.
+
+Preparing the Data
+------------------
+
+For the server to work, CCP4/MAP (models 0, 1, 2 are supported) input data need to be converted into a custom block format. 
+To achieve this, use the ``pack`` application.
+
+- To prepare data from x-ray based methods, use: 
+
+    ```
+    node pack -xray main.ccp4 diff.ccp4 out.mdb
+    ```
+
+- For EM data, use:
+
+    ```
+    node pack -em em.map out.mdb
+    ```
+
+Running the Server
+------------------
+
+- Install production dependencies:
+
+   ```
+   npm install --only=production
+   ```
+
+- Update ``server-config.js`` to link to your data and optionally tweak the other parameters.
+
+- Run it:
+
+    ```
+    node server
+    ```
+
+    In production it is a good idea to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever).
+
+### Local Mode
+
+The program ``local`` in the build folder can be used to query the data without running a http server.
+
+- ``node local`` prints the program usage.
+- ``node local jobs.json`` takes a list of jobs to execute in JSON format. A job entry is defined by this interface:
+
+    ```TypeScript
+    interface JobEntry {
+        source: {
+            filename: string,    
+            name: string,
+            id: string
+        },
+        query: {
+            kind: 'box' | 'cell',
+            space?: 'fractional' | 'cartesian',
+            bottomLeft?: number[],
+            topRight?: number[],
+        }
+        params: {
+            /** Determines the detail level as specified in server-config */
+            detail?: number,
+            /** 
+             * Determines the sampling level:
+             * 1: Original data
+             * 2: Downsampled by factor 1/2
+             * ...
+             * N: downsampled 1/2^(N-1)
+             */
+            forcedSamplingLevel?: number,
+            asBinary: boolean,
+        },
+        outputFolder: string
+    }
+    ```
+
+    Example ``jobs.json`` file content:
+
+    ```TypeScript
+    [{
+        source: {
+            filename: `g:/test/mdb/emd-8116.mdb`,
+            name: 'em',
+            id: '8116',
+        },
+        query: {
+            kind: 'cell'
+        },
+        params: {
+            detail: 4,
+            asBinary: true
+        },
+        outputFolder: 'g:/test/local-test'
+    }]
+    ```
+
+## Navigating the Source Code
+
+The source code is split into 2 mains parts: ``pack`` and ``server``:
+
+- The ``pack`` part provides the means of converting CCP4 files into the internal block format.
+- The ``server`` includes
+  - ``query``: the main part of the server that handles a query. ``execute.ts`` is the "entry point".
+  - ``algebra``: linear, "coordinate", and "box" algebra provides the means for calculations necessary to concent a user query into a menaningful response.
+  - API wrapper that handles the requests.
+
+Consuming the Data 
+==================
+
+The data can be consumed in any (modern) browser using the [CIFTools.js library](https://github.com/dsehnal/CIFTools.js) (or any other piece of code that can read text or binary CIF).
+
+The [Data Format](DataFormat.md) document gives a detailed description of the server response format.
+
+As a reference/example of the server usage, please see the implementation in [LiteMol](https://github.com/dsehnal/LiteMol) ([CIF.ts + Data.ts](https://github.com/dsehnal/LiteMol/tree/master/src/lib/Core/Formats/Density), [UI](https://github.com/dsehnal/LiteMol/tree/master/src/Viewer/Extensions/DensityStreaming)) or in Mol*.
\ No newline at end of file

docs/volume-server/examples.md

+Zika Virus
+==========
+
+![Zika Virus](img/zika_downsampled.png)
+
+1TQN
+====
+
+![1tqn](img/1tqn_downsampled.png)

docs/volume-server/how-it-works.md

+How it works
+============
+
+This document provides a high level overview of how the DensityServer works.
+
+## Overview
+
+- Data is stored in using block layout to reduce the number of disk seeks/reads each query requires.
+- Data is downsampled by ``1/2``, ``1/4``, ``1/8``, ... depending on the size of the input.
+- To keep the server response time/size small, each query is satisfied using the appropriate downsampling level.
+- The server response is encoded using the [BinaryCIF](https://github.com/dsehnal/BinaryCIF) format.
+- The contour level is preserved using relative instead of absolute values.
+
+## Data Layout
+
+To enable efficient access to the 3D data, the density values are stored in a "block level" format. 
+This means that the data is split into ``NxNxN`` blocks (by default ``N=96``, which corresponds to ``96^3 * 4 bytes = 3.375MB`` disk read 
+per block access and provides good size/performance ratio).  This data layout 
+enables to access the data from a hard drive using a bounded number of disk seeks/reads which
+greatly reduces the server latency.
+
+## Downsampling 
+
+- The input is density data with ``[H,K,L]`` number of samples along each axis (i.e. the ``extent`` field in the CCP4 header).
+- To downsample, use the kernel ``C = [1,4,6,4,1]`` (customizable on the source code level) along each axis, because it is "separable":
+
+    ```
+    downsampled[i] = C[0] * source[2 * i - 2] + ... + C[4] * source[2 * i + 2]
+    ```
+
+    The downsampling step is applied in 3 steps:
+
+    ```
+    [H,K,L] => [H/2, K, L] => [H/2, K/2, L] => [H/2, K/2, L/2]
+    ```
+
+    (if the dimension is odd, the value ``(D+1)/2`` is used instead).
+
+- Apply the downsampling step iteratively until the number of samples along the largest dimension is smaller than "block size" (or the smallest dimension has >2 samples).
+
+## Satisfying the query
+
+When the server receives a query for a 3D region, it chooses the the appropriate downsampling level based on the required details so that 
+the number of voxels in the response is small enough. This enables sub-second response time even for the largest of entries.
+
+### Encoding the response
+
+The [BinaryCIF](https://github.com/dsehnal/BinaryCIF) format is used to encode the response. Floating point data are quantized into 1 byte values (256 levels) before being
+sent back to the client. This quantization is performed by splitting the data interval into uniform pieces.
+
+## Preserving the contour level
+
+Downsampling the data results in changing of absolute contour levels. To mitigate this effect, relative values are always used when displaying the data.
+
+- Imagine the input data points are ``A = [-0.3, 2, 0.1, 6, 3, -0.4]``: 
+- Downsampling using every other value results in ``B = [-0.3, 0.1, 3]``.
+- The "range" of the data went from (-0.4, 6) to (-0.3,3).
+- Attempting to use the same absolute contour level on both "data sets" will likely yield very different results.
+- The effect is similar if instead of skipping values they are averaged (or weighted averaged in the case of the ``[1 4 6 4 1]`` kernel) only not as severe.
+- As a result, the "absolute range" of the data changes, some outlier values are lost, but the mean and relative proportions (i.e. deviation ``X`` from mean in ``Y = mean + sigma * X``) are preserved. 
+
+----------------------
+
+## Compression Analysis
+
+- Downsampling: ``i-th`` level (starting from zero) reduces the size by approximate factor ``1/[(2^i)^3]`` (i.e. "cubic" of the frequency).
+- BinaryCIF: CCP4 mode 2 (32 bit floats) is reduced by factor of 4, CCP4 mode 1 (16bit integers) by factor of 2, CCP4 mode 0 (just bytes) is not reduced. This is done by single byte quantization, but smarter than CCP4 mode 0
+- Gzip, from observation:
+  - Gzipping BinaryCIF reduces the size by factor ~2 - ~7 (2 for "dense" data such as x-ray density, 7 for sparse data such such an envelope of a virus)
+  - Gzipping CCP4 reduces the size by 10-25% (be it mode 2 or 0)
+- Applying the downsampling kernel helps with the compression ratios because it smooths out the values.
+
+### Toy example:
+
+```
+Start with 3.5GB compressed density data in the CCP4 mode 2 format (32-bit float for each value)
+    => ~4GB uncompressed CCP4
+    => Downsample by 1/4 => 4GB * (1/4)^3 = 62MB
+    => Convert to BinaryCIF => 62MB / 4 = ~16MB
+    => Gzip: 2 - 8 MB depending on the "density" of the data 
+        (e.g. a viral shell data will be smaller because it is "empty" inside)
+```
\ No newline at end of file

docs/volume-server/response-data-format.md

+Data Format
+===========
+
+This document describes the CIF categories and fields generated by the server.
+
+Query info
+----------
+
+The reponse always contains a data block called ``SERVER`` with this format:
+
+```
+data_SERVER
+#
+_density_server_result.server_version     0.9.0 
+_density_server_result.datetime_utc       '2017-03-09 20:35:45' 
+_density_server_result.guid               f69581b4-6b48-4fa4-861f-c879b4323688 
+_density_server_result.is_empty           no 
+_density_server_result.has_error          no 
+_density_server_result.error              . 
+_density_server_result.query_source_id    xray/1cbs 
+_density_server_result.query_type         box 
+_density_server_result.query_box_type     cartesian 
+_density_server_result.query_box_a[0]     14.555 
+_density_server_result.query_box_a[1]     16.075001 
+_density_server_result.query_box_a[2]     9.848 
+_density_server_result.query_box_b[0]     29.302999 
+_density_server_result.query_box_b[1]     35.737 
+_density_server_result.query_box_b[2]     32.037001 
+```
+
+Query data
+----------
+
+If the query completed successfully with a non-empty result the response will contain one or more data blocks that correpond to the
+"channels" present in the data (e.g. for x-ray data there will be ``2Fo-Fc`` and ``Fo-Fc``) channels.
+
+The format is this:
+
+```
+data_2FO-FC
+#
+_volume_data_3d_info.name                         2Fo-Fc 
+```
+### Axis order
+
+Axis order determines the order of axes of ``origin``, ``dimensions`` and ``sample_count`` fields. It also specifies
+the order of values in ``_volume_data_3d.values``. ``0`` = X axis, ``1`` = Y axis, ``2`` = Z axis.
+
+```
+_volume_data_3d_info.axis_order[0]                0 
+_volume_data_3d_info.axis_order[1]                1 
+_volume_data_3d_info.axis_order[2]                2  
+```
+
+### Origin and dimensions
+
+Specifies the fractional coordinates of the bounding box of the data present in the data block.
+
+```
+_volume_data_3d_info.origin[0]                    -0.5 
+_volume_data_3d_info.origin[1]                    -0.5 
+_volume_data_3d_info.origin[2]                    -0.5 
+_volume_data_3d_info.dimensions[0]                1 
+_volume_data_3d_info.dimensions[1]                1 
+_volume_data_3d_info.dimensions[2]                1 
+```
+
+### Sample rate
+
+Determines how many values of the original input data were collapsed in to 1 value.
+
+```
+_volume_data_3d_info.sample_rate                  8 
+```
+
+### Sample count
+
+Determines how many values in ``_volume_data_3d.values`` are present along each axis (in ``axis_order``).
+
+```
+_volume_data_3d_info.sample_count[0]              96 
+_volume_data_3d_info.sample_count[1]              96 
+_volume_data_3d_info.sample_count[2]              96 
+```
+
+### Spacegroup information
+
+```
+_volume_data_3d_info.spacegroup_number            1 
+_volume_data_3d_info.spacegroup_cell_size[0]      798.72 
+_volume_data_3d_info.spacegroup_cell_size[1]      798.72 
+_volume_data_3d_info.spacegroup_cell_size[2]      798.72 
+_volume_data_3d_info.spacegroup_cell_angles[0]    90 
+_volume_data_3d_info.spacegroup_cell_angles[1]    90 
+_volume_data_3d_info.spacegroup_cell_angles[2]    90 
+```
+
+### Values info
+
+Contains mean, standard deviation, min, and max values for the _entire_ (i.e. not just the slice present in response)
+original and the downsampled data. Both types of values are present 
+so that relative iso-levels can be estimated when sampling changes between queries.
+
+```
+_volume_data_3d_info.mean_source                  0.026747 
+_volume_data_3d_info.mean_sampled                 0.026748 
+_volume_data_3d_info.sigma_source                 1.129252 
+_volume_data_3d_info.sigma_sampled                0.344922 
+_volume_data_3d_info.min_source                   -19.308199 
+_volume_data_3d_info.min_sampled                  -2.692016 
+_volume_data_3d_info.max_source                   26.264214 
+_volume_data_3d_info.max_sampled                  3.533172 
+```
+
+### Values
+
+The values are stored in the ``_volume_data_3d.values`` loop containg ``sample_count[0] * sample_count[1] * sample_count[2]`` values. ``axis_order[0]`` is the axis that changes the fastest, ``axis_order[2]`` is the axis that changes the slowest, same as in the [CCP4 format](http://www.ccp4.ac.uk/html/maplib.html#description)).
+
+```
+loop_
+_volume_data_3d.values
+-0.075391 
+-0.078252 
+-0.078161 
+...
+```
\ No newline at end of file

package.json

@@ -58,6 +58,7 @@
   "devDependencies": {
     "@types/argparse": "^1.0.33",
     "@types/benchmark": "^1.0.31",
+    "@types/compression": "0.0.36",
     "@types/express": "^4.11.1",
     "@types/jest": "^22.2.3",
     "@types/node": "^10.1.1",
@@ -90,6 +91,7 @@
   },
   "dependencies": {
     "argparse": "^1.0.10",
+    "compression": "^1.7.2",
     "express": "^4.16.3",
     "gl": "^4.0.4",
     "immutable": "^4.0.0-rc.9",

src/apps/domain-annotation-server/mapping.ts

@@ -5,7 +5,7 @@
  */
 
 import { Table } from 'mol-data/db'
-import { CIFEncoder, create as createEncoder } from 'mol-io/writer/cif'
+import { EncoderInstance, create as createEncoder } from 'mol-io/writer/cif'
 import * as S from './schemas'
 import { getCategoryInstanceProvider } from './utils'
 
@@ -36,7 +36,7 @@ interface DomainAnnotation {
 }
 type MappingRow = Table.Row<S.mapping>;
 
-function writeDomain(enc: CIFEncoder<any>, domain: DomainAnnotation | undefined) {
+function writeDomain(enc: EncoderInstance, domain: DomainAnnotation | undefined) {
     if (!domain) return;
     enc.writeCategory(getCategoryInstanceProvider(`pdbx_${domain.name}_domain_annotation`, domain.domains));
     enc.writeCategory(getCategoryInstanceProvider(`pdbx_${domain.name}_domain_mapping`, domain.mappings));

src/mol-app/ui/transform/spacefill.tsx

@@ -17,8 +17,8 @@ import { SpacefillUpdate } from 'mol-view/state/transform'
 import { StateContext } from 'mol-view/state/context';
 import { ColorTheme } from 'mol-geo/theme';
 import { Color, ColorNames } from 'mol-util/color';
-import { Query, Queries as Q } from 'mol-model/structure';
-import { Slider } from '../controls/slider';
+// import { Query, Queries as Q } from 'mol-model/structure';
+// import { Slider } from '../controls/slider';
 
 export const ColorThemeInfo = {
     'atom-index': {},

src/mol-io/writer/cif.ts

@@ -13,14 +13,14 @@ import { CategoryDefinition } from './cif/encoder'
 
 export * from './cif/encoder'
 
-export function create(params?: { binary?: boolean, encoderName?: string }) {
+export type EncoderInstance = BinaryCIFEncoder<{}> | TextCIFEncoder<{}>
+
+export function create(params?: { binary?: boolean, encoderName?: string }): EncoderInstance {
     const { binary = false, encoderName = 'mol*' } = params || {};
     return binary ? new BinaryCIFEncoder(encoderName) : new TextCIFEncoder();
 }
 
-type CIFEncoder = BinaryCIFEncoder<{}> | TextCIFEncoder<{}>
-
-export function writeDatabase(encoder: CIFEncoder, name: string, database: Database<Database.Schema>) {
+export function writeDatabase(encoder: EncoderInstance, name: string, database: Database<Database.Schema>) {
     encoder.startDataBlock(name);
     for (const table of database._tableNames) {
         encoder.writeCategory(
@@ -29,7 +29,7 @@ export function writeDatabase(encoder: CIFEncoder, name: string, database: Datab
     }
 }
 
-export function writeDatabaseCollection(encoder: CIFEncoder, collection: DatabaseCollection<Database.Schema>) {
+export function writeDatabaseCollection(encoder: EncoderInstance, collection: DatabaseCollection<Database.Schema>) {
     for (const name of Object.keys(collection)) {
         writeDatabase(encoder, name, collection[name])
     }

src/mol-io/writer/cif/encoder.ts

@@ -70,7 +70,7 @@ export interface CategoryProvider {
     (ctx: any): CategoryInstance
 }
 
-export interface CIFEncoder<T = string | Uint8Array, Context = any> extends Encoder<T> {
+export interface CIFEncoder<T = string | Uint8Array, Context = any> extends Encoder {
     startDataBlock(header: string): void,
     writeCategory(category: CategoryProvider, contexts?: Context[]): void,
     getData(): T

src/mol-io/writer/cif/encoder/binary.ts

@@ -59,8 +59,8 @@ export default class BinaryCIFWriter<Context> implements CIFEncoder<Uint8Array,
         this.dataBlocks = <any>null;
     }
 
-    writeTo(writer: Writer<Uint8Array>) {
-        writer.write(this.encodedData);
+    writeTo(writer: Writer) {
+        writer.writeBinary(this.encodedData);
     }
 
     getData() {

src/mol-io/writer/cif/encoder/text.ts

@@ -48,10 +48,10 @@ export default class TextCIFEncoder<Context> implements Enc.CIFEncoder<string, C
         this.encoded = true;
     }
 
-    writeTo(stream: Writer<string>) {
+    writeTo(stream: Writer) {
         const chunks = StringBuilder.getChunks(this.builder);
         for (let i = 0, _i = chunks.length; i < _i; i++) {
-            stream.write(chunks[i]);
+            stream.writeString(chunks[i]);
         }
     }
 

src/mol-io/writer/encoder.ts

@@ -6,9 +6,9 @@
 
 import Writer from './writer'
 
-interface Encoder<T> {
+interface Encoder {
     encode(): void,
-    writeTo(writer: Writer<T>): void
+    writeTo(writer: Writer): void
 }
 
 export default Encoder
\ No newline at end of file

src/mol-io/writer/writer.ts

@@ -4,8 +4,9 @@
  * @author David Sehnal <david.sehnal@gmail.com>
  */
 
-interface Writer<T> {
-    write(data: T): boolean
+interface Writer {
+    writeString(data: string): boolean,
+    writeBinary(data: Uint8Array): boolean
 }
 
 namespace Writer {

src/servers/volume/CHANGELOG.md

+# 0.9.5
+* Better query response box resolution.
+
+# 0.9.4
+* Support for CCP4 Mode 1.
+* Asking for summary for a non-existing entry will now correctly return 404.
+* Automatically determine max block size for large entries.
+
+# 0.9.3
+* Fixed a bug in CIFTools (BinaryCIF encoding).
+
+# 0.9.2
+* Changed "EM" naming access scheme.
+
+# 0.9.1
+* Removed max fractional dimension checking.
+* Fix a bug in MDB packer.
+
+# 0.9.0
+* Rewrote pretty much everything to add downsampling.
+
+# 0.8.4
+* Add support for converting CCP4 mode 0 maps.
+
+# 0.8.3
+* Allow periodic boundary for P 1 spacegroup.
+
+# 0.8.2
+* Fixed axis ordering issue.
+
+# 0.8.1
+* Updated value encoding.
+
+# 0.8.0
+* Let's call this an initial version.
\ No newline at end of file

src/servers/volume/common/binary-schema.ts

+/**
+ * Copyright (c) 2018 mol* contributors, licensed under MIT, See LICENSE file for more info.
+ *
+ * Taken/adapted from DensityServer (https://github.com/dsehnal/DensityServer)
+ *
+ * @author David Sehnal <david.sehnal@gmail.com>
+ */
+
+import * as UTF8 from 'mol-io/common/utf8'
+
+export type Bool = { kind: 'bool' }
+export type Int = { kind: 'int' }
+export type Float = { kind: 'float' }
+export type String = { kind: 'string' }
+export type Array = { kind: 'array', element: Element }
+export type Prop = { element: Element, prop: string }
+export type Obj = { kind: 'object', props: Prop[] }
+// tslint:disable-next-line:array-type
+export type Element = Bool | Int | Float | String | Array | Obj
+
+export const bool: Bool = { kind: 'bool' };
+export const int: Int = { kind: 'int' };
+export const float: Float = { kind: 'float' };
+export const str: String = { kind: 'string' };
+// tslint:disable-next-line:array-type
+export function array(element: Element): Array { return { kind: 'array', element }; }
+export function obj<T>(schema: ((keyof T) | Element)[][]): Obj {
+    return {
+        kind: 'object',
+        props: schema.map(s => ({
+            element: s[1] as Element,
+            prop: s[0] as string
+        }))
+    };
+}
+
+function byteCount(e: Element, src: any) {
+    let size = 0;
+    switch (e.kind) {
+        case 'bool': size += 1; break;
+        case 'int': size += 4; break;
+        case 'float': size += 8; break;
+        case 'string': size += 4 + UTF8.utf8ByteCount(src); break;
+        case 'array': {
+            size += 4; // array length
+            for (const x of src) {
+                size += byteCount(e.element, x);
+            }
+            break;
+        }
+        case 'object': {
+            for (const p of e.props) {
+                size += byteCount(p.element, src[p.prop]);
+            }
+            break;
+        }
+    }
+    return size;
+}
+
+function writeElement(e: Element, buffer: Buffer, src: any, offset: number) {
+    switch (e.kind) {
+        case 'bool': buffer.writeInt8(src ? 1 : 0, offset); offset += 1; break;
+        case 'int': buffer.writeInt32LE(src | 0, offset); offset += 4; break;
+        case 'float': buffer.writeDoubleLE(+src, offset); offset += 8; break;
+        case 'string': {
+            const val = '' + src;
+            const size = UTF8.utf8ByteCount(val);
+            buffer.writeInt32LE(size, offset);
+            offset += 4; // str len
+            const str = new Uint8Array(size);
+            UTF8.utf8Write(str, 0, val);
+            for (const b of <number[]><any>str) {
+                buffer.writeUInt8(b, offset);
+                offset++;
+            }
+            break;
+        }
+        case 'array': {
+            buffer.writeInt32LE(src.length, offset);
+            offset += 4; // array length
+            for (const x of src) {
+                offset = writeElement(e.element, buffer, x, offset);
+            }
+            break;
+        }
+        case 'object': {
+            for (const p of e.props) {
+                offset = writeElement(p.element, buffer, src[p.prop], offset);
+            }
+            break;
+        }
+    }
+    return offset;
+}
+
+function write(element: Element, src: any) {
+    const size = byteCount(element, src);
+    const buffer = new Buffer(size);
+    writeElement(element, buffer, src, 0);
+    return buffer;
+}
+
+export function encode(element: Element, src: any): Buffer {
+    return write(element, src);
+}
+
+function decodeElement(e: Element, buffer: Buffer, offset: number, target: { value: any }) {
+    switch (e.kind) {
+        case 'bool': target.value = !!buffer.readInt8(offset); offset += 1; break;
+        case 'int': target.value = buffer.readInt32LE(offset); offset += 4; break;
+        case 'float': target.value = buffer.readDoubleLE(offset); offset += 8; break;
+        case 'string': {
+            const size = buffer.readInt32LE(offset);
+            offset += 4; // str len
+            const str = new Uint8Array(size);
+            for (let i = 0; i < size; i++) {
+                str[i] = buffer.readUInt8(offset);
+                offset++;
+            }
+            target.value = UTF8.utf8Read(str, 0, size);
+            break;
+        }
+        case 'array': {
+            const array: any[] = [];
+            const count = buffer.readInt32LE(offset);
+            const element = { value: void 0 };
+            offset += 4;
+            for (let i = 0; i < count; i++) {
+                offset = decodeElement(e.element, buffer, offset, element);
+                array.push(element.value);
+            }
+            target.value = array;
+            break;
+        }
+        case 'object': {
+            const t = Object.create(null);
+            const element = { value: void 0 };
+            for (const p of e.props) {
+                offset = decodeElement(p.element, buffer, offset, element);
+                t[p.prop] = element.value;
+            }
+            target.value = t;
+            break;
+        }
+    }
+    return offset;
+}
+
+export function decode<T>(element: Element, buffer: Buffer, offset?: number) {
+    const target = { value: void 0 as any };
+    decodeElement(element, buffer, offset! | 0, target);
+    return target.value as T;
+}
\ No newline at end of file

src/servers/volume/common/data-format.ts

+/**
+ * Copyright (c) 2018 mol* contributors, licensed under MIT, See LICENSE file for more info.
+ *
+ * Taken/adapted from DensityServer (https://github.com/dsehnal/DensityServer)
+ *
+ * @author David Sehnal <david.sehnal@gmail.com>
+ */
+
+import * as File from './file'
+import * as Schema from './binary-schema'
+
+export type ValueType = 'float32' | 'int8' | 'int16'
+
+export namespace ValueType {
+    export const Float32: ValueType = 'float32';
+    export const Int8: ValueType = 'int8';
+    export const Int16: ValueType = 'int16';
+}
+
+export type ValueArray = Float32Array | Int8Array | Int16Array
+
+export interface Spacegroup {
+    number: number,
+    size: number[],
+    angles: number[],
+    /** Determine if the data should be treated as periodic or not. (e.g. X-ray = periodic, EM = not periodic) */
+    isPeriodic: boolean,
+}
+
+export interface ValuesInfo {
+    mean: number,
+    sigma: number,
+    min: number,
+    max: number
+}
+
+export interface Sampling {
+    byteOffset: number,
+
+    /** How many values along each axis were collapsed into 1 */
+    rate: number,
+    valuesInfo: ValuesInfo[],
+
+    /** Number of samples along each axis, in axisOrder  */
+    sampleCount: number[]
+}
+
+export interface Header {
+    /** Format version number  */
+    formatVersion: string,
+
+    /** Axis order from the slowest to fastest moving, same as in CCP4 */
+    axisOrder: number[],
+
+    /** Origin in fractional coordinates, in axisOrder */
+    origin: number[],
+
+    /** Dimensions in fractional coordinates, in axisOrder */
+    dimensions: number[],
+
+    spacegroup: Spacegroup,
+    channels: string[],
+
+    /** Determines the data type of the values */
+    valueType: ValueType,
+
+    /** The value are stored in blockSize^3 cubes */
+    blockSize: number,
+    sampling: Sampling[]
+}
+
+namespace _schema {
+    const { array, obj, int, bool, float, str } = Schema
+
+    export const schema = obj<Header>([
+        ['formatVersion', str],
+        ['axisOrder', array(int)],
+        ['origin', array(float)],
+        ['dimensions', array(float)],
+        ['spacegroup', obj<Spacegroup>([
+            ['number', int],
+            ['size', array(float)],
+            ['angles', array(float)],
+            ['isPeriodic', bool],
+        ])],
+        ['channels', array(str)],
+        ['valueType', str],
+        ['blockSize', int],
+        ['sampling', array(obj<Sampling>([
+            ['byteOffset', float],
+            ['rate', int],
+            ['valuesInfo', array(obj<ValuesInfo>([
+                ['mean', float],
+                ['sigma', float],
+                ['min', float],
+                ['max', float]
+            ]))],
+            ['sampleCount', array(int)]
+        ]))]
+    ]);
+}
+
+const headerSchema = _schema.schema;
+
+export function getValueByteSize(type: ValueType) {
+    if (type === ValueType.Float32) return 4;
+    if (type === ValueType.Int16) return 2;
+    return 1;
+}
+
+export function createValueArray(type: ValueType, size: number) {
+    switch (type) {
+        case ValueType.Float32: return new Float32Array(new ArrayBuffer(4 * size));
+        case ValueType.Int8: return new Int8Array(new ArrayBuffer(1 * size));
+        case ValueType.Int16: return new Int16Array(new ArrayBuffer(2 * size));
+    }
+    throw Error(`${type} is not a supported value format.`);
+}
+
+export function encodeHeader(header: Header) {
+    return Schema.encode(headerSchema, header);
+}
+
+export async function readHeader(file: number): Promise<{ header: Header, dataOffset: number }> {
+    let { buffer } = await File.readBuffer(file, 0, 4 * 4096);
+    const headerSize = buffer.readInt32LE(0);
+
+    if (headerSize > buffer.byteLength - 4) {
+        buffer = (await File.readBuffer(file, 0, headerSize + 4)).buffer;
+    }
+
+    const header = Schema.decode<Header>(headerSchema, buffer, 4);
+    return { header, dataOffset: headerSize + 4 };
+}
\ No newline at end of file

src/servers/volume/common/file.ts

+/**
+ * Copyright (c) 2018 mol* contributors, licensed under MIT, See LICENSE file for more info.
+ *
+ * Taken/adapted from DensityServer (https://github.com/dsehnal/DensityServer)
+ *
+ * @author David Sehnal <david.sehnal@gmail.com>
+ */
+
+import * as fs from 'fs'
+import * as path from 'path'
+import * as DataFormat from './data-format'
+
+export const IsNativeEndianLittle = new Uint16Array(new Uint8Array([0x12, 0x34]).buffer)[0] === 0x3412;
+
+export async function openRead(filename: string) {
+    return new Promise<number>((res, rej) => {
+        fs.open(filename, 'r', async (err, file) => {
+            if (err) {
+                rej(err);
+                return;
+            }
+
+            try {
+                res(file);
+            } catch (e) {
+                fs.closeSync(file);
+            }
+        })
+    });
+}
+
+export function readBuffer(file: number, position: number, sizeOrBuffer: Buffer | number, size?: number, byteOffset?: number): Promise<{ bytesRead: number, buffer: Buffer }> {
+    return new Promise((res, rej) => {
+        if (typeof sizeOrBuffer === 'number') {
+            let buff = new Buffer(new ArrayBuffer(sizeOrBuffer));
+            fs.read(file, buff, 0, sizeOrBuffer, position, (err, bytesRead, buffer) => {
+                if (err) {
+                    rej(err);
+                    return;
+                }
+                res({ bytesRead, buffer });
+            });
+        } else {
+            if (size === void 0) {
+                rej('readBuffer: Specify size.');
+                return;
+            }
+
+            fs.read(file, sizeOrBuffer, byteOffset ? +byteOffset : 0, size, position, (err, bytesRead, buffer) => {
+                if (err) {
+                    rej(err);
+                    return;
+                }
+                res({ bytesRead, buffer });
+            });
+        }
+    })
+}
+
+export function writeBuffer(file: number, position: number, buffer: Buffer, size?: number): Promise<number> {
+    return new Promise<number>((res, rej) => {
+        fs.write(file, buffer, 0, size !== void 0 ? size : buffer.length, position, (err, written) => {
+            if (err) rej(err);
+            else res(written);
+        })
+    })
+}
+
+function makeDir(path: string, root?: string): boolean {
+    let dirs = path.split(/\/|\\/g),
+        dir = dirs.shift();
+
+    root = (root || '') + dir + '/';
+
+    try { fs.mkdirSync(root); }
+    catch (e) {
+        if (!fs.statSync(root).isDirectory()) throw new Error(e);
+    }
+
+    return !dirs.length || makeDir(dirs.join('/'), root);
+}
+
+export function exists(filename: string) {
+    return fs.existsSync(filename);
+}
+
+export function createFile(filename: string) {
+    return new Promise<number>((res, rej) => {
+        if (fs.existsSync(filename)) fs.unlinkSync(filename);
+        makeDir(path.dirname(filename));
+        fs.open(filename, 'w', (err, file) => {
+            if (err) rej(err);
+            else res(file);
+        })
+    });
+}
+
+const __emptyFunc = function () { };
+export function close(file: number | undefined) {
+    try {
+        if (file !== void 0) fs.close(file, __emptyFunc);
+    } catch (e) {
+
+    }
+}
+
+const smallBuffer = new Buffer(8);
+export async function writeInt(file: number, value: number, position: number) {
+    smallBuffer.writeInt32LE(value, 0);
+    await writeBuffer(file, position, smallBuffer, 4);
+}
+
+export interface TypedArrayBufferContext {
+    type: DataFormat.ValueType,
+    elementByteSize: number,
+    readBuffer: Buffer,
+    valuesBuffer: Uint8Array,
+    values: DataFormat.ValueArray
+}
+
+function getElementByteSize(type: DataFormat.ValueType) {
+    if (type === DataFormat.ValueType.Float32) return 4;
+    if (type === DataFormat.ValueType.Int16) return 2;
+    return 1;
+}
+
+function makeTypedArray(type: DataFormat.ValueType, buffer: ArrayBuffer): DataFormat.ValueArray {
+    if (type === DataFormat.ValueType.Float32) return new Float32Array(buffer);
+    if (type === DataFormat.ValueType.Int16) return new Int16Array(buffer);
+    return new Int8Array(buffer);
+}
+
+export function createTypedArrayBufferContext(size: number, type: DataFormat.ValueType): TypedArrayBufferContext {
+    let elementByteSize = getElementByteSize(type);
+    let arrayBuffer = new ArrayBuffer(elementByteSize * size);
+    let readBuffer = new Buffer(arrayBuffer);
+    let valuesBuffer = IsNativeEndianLittle ? arrayBuffer : new ArrayBuffer(elementByteSize * size);
+    return {
+        type,
+        elementByteSize,
+        readBuffer,
+        valuesBuffer: new Uint8Array(valuesBuffer),
+        values: makeTypedArray(type, valuesBuffer)
+    };
+}
+
+function flipByteOrder(source: Buffer, target: Uint8Array, byteCount: number, elementByteSize: number, offset: number) {
+    for (let i = 0, n = byteCount; i < n; i += elementByteSize) {
+        for (let j = 0; j < elementByteSize; j++) {
+            target[offset + i + elementByteSize - j - 1] = source[offset + i + j];
+        }
+    }
+}
+
+export async function readTypedArray(ctx: TypedArrayBufferContext, file: number, position: number, count: number, valueOffset: number, littleEndian?: boolean) {
+    let byteCount = ctx.elementByteSize * count;
+    let byteOffset = ctx.elementByteSize * valueOffset;
+
+    await readBuffer(file, position, ctx.readBuffer, byteCount, byteOffset);
+    if (ctx.elementByteSize > 1 && ((littleEndian !== void 0 && littleEndian !== IsNativeEndianLittle) || !IsNativeEndianLittle)) {
+        // fix the endian 
+        flipByteOrder(ctx.readBuffer, ctx.valuesBuffer, byteCount, ctx.elementByteSize, byteOffset);
+    }
+    return ctx.values;
+}
+
+export function ensureLittleEndian(source: Buffer, target: Buffer, byteCount: number, elementByteSize: number, offset: number) {
+    if (IsNativeEndianLittle) return;
+    if (!byteCount || elementByteSize <= 1) return;
+    flipByteOrder(source, target, byteCount, elementByteSize, offset);
+}
\ No newline at end of file