diff --git a/docs/model-server/readme.md b/docs/model-server/readme.md index f091200c855656f1fb7599369fb6fc6bc679b592..dc73ab041adddd05d9dff494a6d82c966e41d55e 100644 --- a/docs/model-server/readme.md +++ b/docs/model-server/readme.md @@ -6,133 +6,64 @@ Model Server is a tool for preprocessing and querying macromolecular structure d Installing and Running ===================== -Getting the code (use node 8+): +Requires nodejs 8+. + +## From GitHub + ``` git clone https://github.com/molstar/molstar npm install ``` -Customize configuration at ``src/server/model/config.ts`` to point to your data and which custom properties to include (see the [Custom Properties](#custom-properties) section). Alternatively, the config can be edited in the compiled version in ``build/node_modules/servers/model/config.js``. - -Afterwards, build the project: +Afterwards, build the project source: ``` -npm run build +npm run build-tsc ``` -(or run watch mode for automatic rebuilds: ``npm run watch``) +and run the server by -Running the server locally for testing: ``` -npm run model-server -``` -or -``` -node build/node_modules/servers/model/server +node lib/servers/model/server/server ``` -In production it is a good idea to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever). +## From NPM +``` +npm install molstar +./model-server +``` -## Memory issues +(or ``node node_modules\.bin\model-server`` in Windows). -Sometimes nodejs might run into problems with memory. This is usually resolved by adding the ``--max-old-space-size=8192`` parameter. +The NPM package contains all the tools mentioned here as "binaries": -Preprocessor -============ +- ``model-server`` +- ``model-server-query`` +- ``model-server-preprocess`` -The preprocessor application allows to add custom data to CIF files and/or convert CIF to BinaryCIF. See the [Custom Properties](#custom-properties) section for providing custom properties. -## Usage +### Production use -The app works in two modes: single files and folders. +In production it is required to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever). -Single files: -``` -node build\node_modules\servers\model\preprocess -i input.cif [-oc output.cif] [-ob output.bcif] [--cfg config.json] -``` +### Memory issues -Folder: -``` -node build\node_modules\servers\model\preprocess -fin input_folder [-foc output_cif_folder] [-fob output_bcif_folder] [--cfg config.json] -``` +Sometimes nodejs might run into problems with memory. This is usually resolved by adding the ``--max-old-space-size=8192`` parameter. -## Config - -The config speficies the maximum number of processes to use (in case of folder processing) and defines sources and parameters for custom properties. - -Example: -```json -{ - "numProcesses": 4, - "customProperties": { - "sources": [ - "./properties/pdbe" - ], - "params": { - "PDBe": { - "UseFileSource": false, - "API": { - "residuewise_outlier_summary": "https://www.ebi.ac.uk/pdbe/api/validation/residuewise_outlier_summary/entry", - "preferred_assembly": "https://www.ebi.ac.uk/pdbe/api/pdb/entry/summary", - "struct_ref_domain": "https://www.ebi.ac.uk/pdbe/api/mappings/sequence_domains" - } - } - } - } -} -``` +## Preprocessor -Custom Properties -================= +The preprocessor application allows to add custom data to CIF files and/or convert CIF to BinaryCIF. ``node lib/servers/model/preprocess`` or ``model-server-preprocess`` binary from the NPM package. -It is possible to provide property descriptors that transform data to internal representation and define how it should be exported into one or mode CIF categories. Examples of this are located in the ``mol-model-props`` module and are linked to the server in the config and ``servers/model/properties``. -Local Mode -========== +## Local Mode -The server can be run in local/file based mode: +The server can be run in local/file based mode using ``node lib/servers/model/query`` (``model-server-query`` binary from the NPM package). -``` -node build/node_modules/servers/model/server jobs.json -``` +Custom Properties +================= -where ``jobs.json`` is an array of - -```ts -type LocalInput = { - input: string, - output: string, - query: QueryName, - modelNums?: number[], - params?: any, - binary?: boolean -}[] -``` +This feature is still in development. -For example - -```json -[ - { - "input": "c:/test/quick/1tqn.cif", - "output": "c:/test/quick/localapi/1tqn_full.cif", - "query": "full" - }, - { - "input": "c:/test/quick/1tqn.cif", - "output": "c:/test/quick/localapi/1tqn_full.bcif", - "query": "full", - "params": {} - }, - { - "input": "c:/test/quick/1cbs_updated.cif", - "output": "c:/test/quick/localapi/1cbs_ligint.cif", - "query": "residueInteraction", - "params": { - "atom_site": { "label_comp_id": "REA" } - } - } -] -``` \ No newline at end of file +It is possible to provide property descriptors that transform data to internal representation and define how it should be exported into one or mode CIF categories. Examples of this are located in the ``mol-model-props`` module and are linked to the server in the config and ``servers/model/properties``. \ No newline at end of file diff --git a/docs/volume-server/README.md b/docs/volume-server/README.md index 91741342c8fc5a5cba0da2b6b4985d6f1359013b..ff4d6437ea39d1d3b802a5bd70c0f8affeae5af3 100644 --- a/docs/volume-server/README.md +++ b/docs/volume-server/README.md @@ -7,107 +7,64 @@ It uses the text based CIF and BinaryCIF formats to deliver the data to the clie For quick info about the benefits of using the server, check out the [examples](examples.md). -Installing the Server +Installing and Running ===================== -- Install [Node.js](https://nodejs.org/en/) (tested on Node 6.* and 7.*; x64 version is strongly preferred). -- Get the code. -- Prepare the data. -- Run the server. +Requires nodejs 8+. -Preparing the Data ------------------- +## From GitHub + +``` +git clone https://github.com/molstar/molstar +npm install +``` + +Afterwards, build the project source: + +``` +npm run build-tsc +``` + +and run the server by + +``` +node lib/servers/volume/server +``` + +## From NPM + +``` +npm install molstar +./volume-server +``` + +(or ``node node_modules\.bin\volume-server`` in Windows). + +The NPM package contains all the tools mentioned here as "binaries": + +- ``volume-server`` +- ``volume-server-pack`` +- ``volume-server-query`` + + +### Production use + +In production it is required to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever). + + +### Memory issues + +Sometimes nodejs might run into problems with memory. This is usually resolved by adding the ``--max-old-space-size=8192`` parameter. + + +## Preparing the Data For the server to work, CCP4/MAP (models 0, 1, 2 are supported) input data need to be converted into a custom block format. -To achieve this, use the ``pack`` application. - -- To prepare data from x-ray based methods, use: - - ``` - node pack -xray main.ccp4 diff.ccp4 out.mdb - ``` - -- For EM data, use: - - ``` - node pack -em em.map out.mdb - ``` - -Running the Server ------------------- - -- Install production dependencies: - - ``` - npm install --only=production - ``` - -- Update ``server-config.js`` to link to your data and optionally tweak the other parameters. - -- Run it: - - ``` - node server - ``` - - In production it is a good idea to use a service that will keep the server running, such as [forever.js](https://github.com/foreverjs/forever). - -### Local Mode - -The program ``local`` in the build folder can be used to query the data without running a http server. - -- ``node local`` prints the program usage. -- ``node local jobs.json`` takes a list of jobs to execute in JSON format. A job entry is defined by this interface: - - ```TypeScript - interface JobEntry { - source: { - filename: string, - name: string, - id: string - }, - query: { - kind: 'box' | 'cell', - space?: 'fractional' | 'cartesian', - bottomLeft?: number[], - topRight?: number[], - } - params: { - /** Determines the detail level as specified in server-config */ - detail?: number, - /** - * Determines the sampling level: - * 1: Original data - * 2: Downsampled by factor 1/2 - * ... - * N: downsampled 1/2^(N-1) - */ - forcedSamplingLevel?: number, - asBinary: boolean, - }, - outputFolder: string - } - ``` - - Example ``jobs.json`` file content: - - ```TypeScript - [{ - source: { - filename: `g:/test/mdb/emd-8116.mdb`, - name: 'em', - id: '8116', - }, - query: { - kind: 'cell' - }, - params: { - detail: 4, - asBinary: true - }, - outputFolder: 'g:/test/local-test' - }] - ``` +To achieve this, use the ``pack`` application (``node lib/servers/volume/pack`` or ``volume-server-pack`` binary from the NPM package). + +## Local Mode + +The program ``lib/servers/volume/pack`` (``volume-server-query`` in NPM package) can be used to query the data without running a http server. ## Navigating the Source Code @@ -122,8 +79,8 @@ The source code is split into 2 mains parts: ``pack`` and ``server``: Consuming the Data ================== -The data can be consumed in any (modern) browser using the [CIFTools.js library](https://github.com/dsehnal/CIFTools.js) (or any other piece of code that can read text or binary CIF). +The data can be consumed in any (modern) browser using the [ciftools library](https://github.com/molstar/ciftools) (or any other piece of code that can read text or binary CIF). The [Data Format](DataFormat.md) document gives a detailed description of the server response format. -As a reference/example of the server usage, please see the implementation in [LiteMol](https://github.com/dsehnal/LiteMol) ([CIF.ts + Data.ts](https://github.com/dsehnal/LiteMol/tree/master/src/lib/Core/Formats/Density), [UI](https://github.com/dsehnal/LiteMol/tree/master/src/Viewer/Extensions/DensityStreaming)) or in Mol*. \ No newline at end of file +As a reference/example of the server usage is available in Mol* ``mol-plugin`` module. \ No newline at end of file diff --git a/package.json b/package.json index 88568d2470f3db4e04bbe4df623f8d09ddc4d397..f41dff10479b9e7a4e66a7bf7a6f57efdfa012ac 100644 --- a/package.json +++ b/package.json @@ -26,7 +26,7 @@ "serve": "http-server -p 1338", "model-server": "node lib/servers/model/server.js", "model-server-watch": "nodemon --watch lib lib/servers/model/server.js", - "volume-server": "node lib/servers/volume/server.js --idMap em 'test/${id}.mdb' --defaultPort 1336", + "volume-server-test": "node lib/servers/volume/server.js --idMap em 'test/${id}.mdb' --defaultPort 1336", "plugin-state": "node lib/servers/plugin-state/index.js", "preversion": "npm run test", "postversion": "git push && git push --tags", @@ -35,6 +35,14 @@ "files": [ "lib/" ], + "bin": { + "model-server": "lib/servers/model/server.js", + "model-server-query": "lib/servers/model/local.js", + "model-server-preprocess": "lib/servers/model/preprocess.js", + "volume-server": "lib/servers/volume/server.js", + "volume-server-query": "lib/servers/volume/query.js", + "volume-server-pack": "lib/servers/volume/pack.js" + }, "nodemonConfig": { "ignoreRoot": [ "./node_modules", diff --git a/src/servers/model/config.ts b/src/servers/model/config.ts index cbc6bf8b69ad4c60222fa956b51fbc9ffbf6afa3..68fbb6168649723faeb5a4bf96296305523ce9c9 100644 --- a/src/servers/model/config.ts +++ b/src/servers/model/config.ts @@ -1,18 +1,19 @@ /** - * Copyright (c) 2018-2019 mol* contributors, licensed under MIT, See LICENSE file for more info. + * Copyright (c) 2018-2020 mol* contributors, licensed under MIT, See LICENSE file for more info. * * @author David Sehnal <david.sehnal@gmail.com> */ +import * as argparse from 'argparse' +import { ObjectKeys } from '../../mol-util/type-helpers' +import VERSION from './version' +import * as fs from 'fs' +import { ModelPropertyProviderConfig } from './property-provider'; + const DefaultModelServerConfig = { - /** - * Determine if and how long to cache entries after a request. - */ - cacheParams: { - useCache: true, - maxApproximateSizeInBytes: 2 * 1014 * 1024 * 1024, // 2 GB - entryTimeoutInMs: 10 * 60 * 1000 // 10 minutes - }, + // 0 for off + cacheMaxSizeInBytes: 2 * 1014 * 1024 * 1024, // 2 GB + cacheEntryTimeoutMs: 10 * 60 * 1000, // 10 minutes /** * Node (V8) sometimes exhibits GC related issues that significantly slow down the execution @@ -22,13 +23,11 @@ const DefaultModelServerConfig = { * For this to work, the server must be run using a deamon (i.e. forever.js on Linux * or IISnode on Windows) so that the server is automatically restarted when the shutdown happens. */ - shutdownParams: { - // 0 for off, server will shut down after this amount of minutes. - timeoutMinutes: 24 * 60 /* a day */, - // modifies the shutdown timer by +/- timeoutVarianceMinutes (to avoid multiple instances shutting at the same time) - timeoutVarianceMinutes: 60 - }, + // 0 for off, server will shut down after this amount of minutes. + shutdownTimeoutMinutes: 24 * 60, /* a day */ + // modifies the shutdown timer by +/- timeoutVarianceMinutes (to avoid multiple instances shutting at the same time) + shutdownTimeoutVarianceMinutes: 60, defaultPort: 1337, @@ -36,13 +35,13 @@ const DefaultModelServerConfig = { * Specify the prefix of the API, i.e. * <host>/<apiPrefix>/<API queries> */ - appPrefix: '/ModelServer', + apiPrefix: '/ModelServer', /** * The maximum time the server dedicates to executing a query. * Does not include the time it takes to read and export the data. */ - maxQueryTimeInMs: 5 * 1000, + queryTimeoutMs: 5 * 1000, /** Maximum number of requests before "server busy" */ maxQueueLength: 30, @@ -51,7 +50,7 @@ const DefaultModelServerConfig = { * Provide a property config or a path a JSON file with the config. */ // TODO: finish customProperty support - customProperties: <import('./property-provider').ModelPropertyProviderConfig | string>{ + customProperties: <ModelPropertyProviderConfig | string>{ sources: [ // 'pdbe', // 'rcsb', @@ -91,26 +90,164 @@ const DefaultModelServerConfig = { * * /static query uses 'pdb-cif' and 'pdb-bcif' source names. */ - fileMapping: [ + sourceMap: [ ['pdb-cif', 'e:/test/quick/${id}_updated.cif'], // ['pdb-bcif', 'e:/test/quick/${id}.bcif'], ] as [string, string][] }; +export let mapSourceAndIdToFilename: (source: string, id: string) => string = () => { + throw new Error('call setupConfig & validateConfigAndSetupSourceMap to initialize this function'); +} + +function addServerArgs(parser: argparse.ArgumentParser) { + parser.addArgument([ '--apiPrefix' ], { + defaultValue: DefaultModelServerConfig.apiPrefix, + metavar: 'PREFIX', + help: `Specify the prefix of the API, i.e. <host>/<apiPrefix>/<API queries>` + }); + parser.addArgument([ '--defaultPort' ], { + defaultValue: DefaultModelServerConfig.defaultPort, + metavar: 'PORT', + type: 'int', + help: `Specify the port the server is running on` + }); + parser.addArgument([ '--cacheMaxSizeInBytes' ], { + defaultValue: DefaultModelServerConfig.cacheMaxSizeInBytes, + metavar: 'CACHE_SIZE', + type: 'int', + help: `0 for off.` + }); + parser.addArgument([ '--cacheEntryTimeoutMs' ], { + defaultValue: DefaultModelServerConfig.cacheEntryTimeoutMs, + metavar: 'CACHE_TIMEOUT', + type: 'int', + help: `Specify in ms how long to keep entries in cache.` + }); + parser.addArgument([ '--queryTimeoutMs' ], { + defaultValue: DefaultModelServerConfig.queryTimeoutMs, + metavar: 'QUERY_TIMEOUT', + type: 'int', + help: `The maximum time the server dedicates to executing a query in ms.\nDoes not include the time it takes to read and export the data.` + }); + parser.addArgument([ '--shutdownTimeoutMinutes' ], { + defaultValue: DefaultModelServerConfig.shutdownTimeoutMinutes, + metavar: 'TIME', + type: 'int', + help: `0 for off, server will shut down after this amount of minutes.` + }); + parser.addArgument([ '--shutdownTimeoutVarianceMinutes' ], { + defaultValue: DefaultModelServerConfig.shutdownTimeoutVarianceMinutes, + metavar: 'VARIANCE', + type: 'int', + help: `modifies the shutdown timer by +/- timeoutVarianceMinutes (to avoid multiple instances shutting at the same time)` + }); + parser.addArgument([ '--defaultSource' ], { + defaultValue: DefaultModelServerConfig.defaultSource, + metavar: 'DEFAULT_SOURCE', + help: `modifies which 'sourceMap' source to use by default` + }); + parser.addArgument([ '--sourceMap' ], { + nargs: 2, + action: 'append', + metavar: ['SOURCE', 'PATH'] as any, + help: [ + 'Map `id`s for a `source` to a file path.', + 'Example: pdb-bcif \'../../data/bcif/${id}.bcif\' ', + 'JS expressions can be used inside ${}, e.g. \'${id.substr(1, 2)}/${id}.mdb\'', + 'Can be specified multiple times.', + 'The `SOURCE` variable (e.g. `pdb-bcif`) is arbitrary and depends on how you plan to use the server.' + ].join('\n'), + }); +} + export type ModelServerConfig = typeof DefaultModelServerConfig -export const ModelServerConfig = DefaultModelServerConfig +export const ModelServerConfig = { ...DefaultModelServerConfig } -export let mapSourceAndIdToFilename: (source: string, id: string) => string = () => { - throw new Error('call setupConfig to initialize this function'); +export const ModelServerConfigTemplate: ModelServerConfig = { + ...DefaultModelServerConfig, + defaultSource: 'pdb-bcif', + sourceMap: [ + ['pdb-bcif', './path-to-binary-cif/${id.substr(1, 2)}/${id}.bcif'], + ['pdb-cif', './path-to-text-cif/${id.substr(1, 2)}/${id}.cif'], + ['pdb-updated', './path-to-updated-cif/${id}.bcif'] + ] as [string, string][] } -export function setupConfig(cfg?: ModelServerConfig) { - if (cfg) Object.assign(ModelServerConfig, cfg); - if (!ModelServerConfig.fileMapping) return; +// TODO: include once properly supported +delete ModelServerConfigTemplate.customProperties +interface ServerJsonConfig { + cfg?: string, + printCfg?: any, + cfgTemplate?: any +} + +function addJsonConfigArgs(parser: argparse.ArgumentParser) { + parser.addArgument(['--cfg'], { + help: [ + 'JSON config file path', + 'If a property is not specified, cmd line param/OS variable/default value are used.' + ].join('\n'), + required: false + }); + parser.addArgument(['--printCfg'], { help: 'Print current config for validation and exit.', required: false, nargs: 0 }); + parser.addArgument(['--cfgTemplate'], { help: 'Prints default JSON config template to be modified and exits.', required: false, nargs: 0 }); +} + +function setConfig(config: ModelServerConfig) { + for (const k of ObjectKeys(ModelServerConfig)) { + if (config[k] !== void 0) (ModelServerConfig as any)[k] = config[k]; + } +} + +function validateConfigAndSetupSourceMap() { + if (!ModelServerConfig.sourceMap || ModelServerConfig.sourceMap.length === 0) { + throw new Error(`Please provide 'sourceMap' configuration. See [-h] for available options.`); + } + mapSourceAndIdToFilename = new Function('source', 'id', [ 'switch (source.toLowerCase()) {', - ...ModelServerConfig.fileMapping.map(([source, path]) => `case '${source.toLowerCase()}': return \`${path}\`;`), + ...ModelServerConfig.sourceMap.map(([source, path]) => `case '${source.toLowerCase()}': return \`${path}\`;`), '}', ].join('\n')) as any; +} + +function parseConfigArguments() { + const parser = new argparse.ArgumentParser({ + version: VERSION, + addHelp: true, + description: `ModelServer ${VERSION}, (c) 2018-2020, Mol* contributors` + }); + addJsonConfigArgs(parser); + addServerArgs(parser); + return parser.parseArgs() as ModelServerConfig & ServerJsonConfig; +} + +export function configureServer() { + const config = parseConfigArguments(); + + if (config.cfgTemplate !== null) { + console.log(JSON.stringify(ModelServerConfigTemplate, null, 2)); + process.exit(0); + } + + try { + setConfig(config) // sets the config for global use + + if (config.cfg) { + const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as ModelServerConfig; + setConfig(cfg); + } + + if (config.printCfg !== null) { + console.log(JSON.stringify(ModelServerConfig, null, 2)); + process.exit(0); + } + + validateConfigAndSetupSourceMap(); + } catch (e) { + console.error('' + e); + process.exit(1); + } } \ No newline at end of file diff --git a/src/servers/model/local.ts b/src/servers/model/query.ts similarity index 96% rename from src/servers/model/local.ts rename to src/servers/model/query.ts index 772c9dafd0263392827c17f49c68636e739ea3ac..6bd54387a871a554bec7d9d1d5457d31db6b3eb0 100644 --- a/src/servers/model/local.ts +++ b/src/servers/model/query.ts @@ -8,7 +8,7 @@ import * as fs from 'fs' import Version from './version'; import { LocalInput, runLocal } from './server/api-local'; -console.log(`Mol* ModelServer (${Version}), (c) 2018 Mol* authors`); +console.log(`Mol* ModelServer (${Version}), (c) 2018-2020 Mol* authors`); console.log(``); let exampleWorkload: LocalInput = [{ diff --git a/src/servers/model/server.ts b/src/servers/model/server.ts index 4dde6c496ace347c505afb92cb560f5e4b435ed9..03e4189fcefcbe50fbee4372e39899f95688763c 100644 --- a/src/servers/model/server.ts +++ b/src/servers/model/server.ts @@ -4,25 +4,23 @@ * @author David Sehnal <david.sehnal@gmail.com> */ -import * as express from 'express' import * as compression from 'compression' -import * as fs from 'fs' -import * as argparse from 'argparse' -import { ModelServerConfig as ServerConfig, setupConfig, ModelServerConfig } from './config' -import { ConsoleLogger } from '../../mol-util/console-logger'; -import { PerformanceMonitor } from '../../mol-util/performance-monitor'; -import { initWebApi } from './server/api-web'; +import * as express from 'express' +import { ConsoleLogger } from '../../mol-util/console-logger' +import { PerformanceMonitor } from '../../mol-util/performance-monitor' +import { configureServer, ModelServerConfig as ServerConfig } from './config' +import { initWebApi } from './server/api-web' import Version from './version' function setupShutdown() { - if (ServerConfig.shutdownParams.timeoutVarianceMinutes > ServerConfig.shutdownParams.timeoutMinutes) { + if (ServerConfig.shutdownTimeoutVarianceMinutes > ServerConfig.shutdownTimeoutMinutes) { ConsoleLogger.log('Server', 'Shutdown timeout variance is greater than the timer itself, ignoring.'); } else { let tVar = 0; - if (ServerConfig.shutdownParams.timeoutVarianceMinutes > 0) { - tVar = 2 * (Math.random() - 0.5) * ServerConfig.shutdownParams.timeoutVarianceMinutes; + if (ServerConfig.shutdownTimeoutVarianceMinutes > 0) { + tVar = 2 * (Math.random() - 0.5) * ServerConfig.shutdownTimeoutVarianceMinutes; } - let tMs = (ServerConfig.shutdownParams.timeoutMinutes + tVar) * 60 * 1000; + let tMs = (ServerConfig.shutdownTimeoutMinutes + tVar) * 60 * 1000; console.log(`----------------------------------------------------------------------------`); console.log(` The server will shut down in ${PerformanceMonitor.format(tMs)} to prevent slow performance.`); @@ -41,22 +39,7 @@ function setupShutdown() { } } -const cmdParser = new argparse.ArgumentParser({ - addHelp: true -}); - -cmdParser.addArgument(['--cfg'], { help: 'Config file path.', required: false }); -cmdParser.addArgument(['--printcfg'], { help: 'Prints out current config and exits.', required: false, nargs: 0 }); - -interface CmdArgs { - cfg?: string, - printcfg?: boolean -} - -const cmdArgs = cmdParser.parseArgs() as CmdArgs; - -const cfg = cmdArgs.cfg ? JSON.parse(fs.readFileSync(cmdArgs.cfg, 'utf8')) : void 0; -setupConfig(cfg); +configureServer(); function startServer() { let app = express(); @@ -73,12 +56,8 @@ function startServer() { console.log(``); } -if (cmdArgs.printcfg !== null) { - console.log(JSON.stringify(ModelServerConfig, null, 2)); -} else { - startServer(); +startServer(); - if (ServerConfig.shutdownParams && ServerConfig.shutdownParams.timeoutMinutes > 0) { - setupShutdown(); - } +if (ServerConfig.shutdownTimeoutMinutes > 0) { + setupShutdown(); } \ No newline at end of file diff --git a/src/servers/model/server/api-schema.ts b/src/servers/model/server/api-schema.ts index 86fc79c407095bbb47f3a5ea6f5e807eb2e7f9da..4e44fed9fda6f01ccfa32bf65455d9bb2892096c 100644 --- a/src/servers/model/server/api-schema.ts +++ b/src/servers/model/server/api-schema.ts @@ -44,7 +44,7 @@ export function getApiSchema() { function getPaths() { const ret: any = {}; for (const { name, definition } of QueryList) { - ret[`${ServerConfig.appPrefix}/v1/{id}/${name}`] = getQueryInfo(definition); + ret[`${ServerConfig.apiPrefix}/v1/{id}/${name}`] = getQueryInfo(definition); } return ret; } diff --git a/src/servers/model/server/api-web.ts b/src/servers/model/server/api-web.ts index 2de281f048c7ea21d02adf6d16c9679e2a329b65..8ad9a70c2ba22f8fb6b354e768c559b1187277da 100644 --- a/src/servers/model/server/api-web.ts +++ b/src/servers/model/server/api-web.ts @@ -18,7 +18,7 @@ import { getApiSchema, shortcutIconLink } from './api-schema'; import { swaggerUiAssetsHandler, swaggerUiIndexHandler } from '../../common/swagger-ui'; function makePath(p: string) { - return Config.appPrefix + '/' + p; + return Config.apiPrefix + '/' + p; } function wrapResponse(fn: string, res: express.Response) { @@ -187,7 +187,7 @@ export function initWebApi(app: express.Express) { app.use(makePath(''), swaggerUiAssetsHandler()); app.get(makePath(''), swaggerUiIndexHandler({ openapiJsonUrl: makePath('openapi.json'), - apiPrefix: Config.appPrefix, + apiPrefix: Config.apiPrefix, title: 'ModelServer API', shortcutIconLink })); diff --git a/src/servers/model/server/cache.ts b/src/servers/model/server/cache.ts index b2978044db48d6ccfecf9a8e441b68c6188ff9c9..365923c17bf1b093fd397f679f07ef9cbb279629 100644 --- a/src/servers/model/server/cache.ts +++ b/src/servers/model/server/cache.ts @@ -48,7 +48,7 @@ export class Cache<T> { private refresh(e: CacheNode<T>) { this.clearTimeout(e); - e.value.timeoutId = setTimeout(() => this.expireNode(e), ServerConfig.cacheParams.entryTimeoutInMs); + e.value.timeoutId = setTimeout(() => this.expireNode(e), ServerConfig.cacheEntryTimeoutMs); this.entries.remove(e); this.entries.addFirst(e.value); } @@ -74,7 +74,7 @@ export class Cache<T> { if (this.entryMap.has(key)) this.dispose(this.entryMap.get(key)!); - if (ServerConfig.cacheParams.maxApproximateSizeInBytes < this.approximateSize + approximateSize) { + if (ServerConfig.cacheMaxSizeInBytes < this.approximateSize + approximateSize) { if (this.entries.last) this.dispose(this.entries.last); } diff --git a/src/servers/model/server/query.ts b/src/servers/model/server/query.ts index 17336e75d355238844b9e9d3b500690a056c7bb7..942f854e9ffd436e39cd4e5a86d5201238415781 100644 --- a/src/servers/model/server/query.ts +++ b/src/servers/model/server/query.ts @@ -57,7 +57,7 @@ export async function resolveJob(job: Job): Promise<CifWriter.Encoder<any>> { const queries = structures.map(s => job.queryDefinition.query(job.normalizedParams, s)); const result: Structure[] = []; for (let i = 0; i < structures.length; i++) { - const s = await StructureSelection.unionStructure(StructureQuery.run(queries[i], structures[i], { timeoutMs: Config.maxQueryTimeInMs })) + const s = await StructureSelection.unionStructure(StructureQuery.run(queries[i], structures[i], { timeoutMs: Config.queryTimeoutMs })) if (s.elementCount > 0) result.push(s); } perf.end('query'); @@ -112,7 +112,7 @@ function doError(job: Job, e: any) { return encoder; } -const maxTime = Config.maxQueryTimeInMs; +const maxTime = Config.queryTimeoutMs; export function abortingObserver(p: Progress) { if (now() - p.root.progress.startedTime > maxTime) { p.requestAbort(`Exceeded maximum allowed time for a query (${maxTime}ms)`); diff --git a/src/servers/model/server/structure-wrapper.ts b/src/servers/model/server/structure-wrapper.ts index 148f3e35aff67a18608c9145de65c5c95b83650a..b9c6a425134c9ea501ee07e94e11b8a6e569bcac 100644 --- a/src/servers/model/server/structure-wrapper.ts +++ b/src/servers/model/server/structure-wrapper.ts @@ -49,12 +49,12 @@ export interface StructureWrapper { } export async function createStructureWrapperFromJob(job: Job, propertyProvider: ModelPropertiesProvider | undefined, allowCache = true): Promise<StructureWrapper> { - if (allowCache && Config.cacheParams.useCache) { + if (allowCache && Config.cacheMaxSizeInBytes > 0) { const ret = StructureCache.get(job.key); if (ret) return ret; } const ret = await readStructureWrapper(job.key, job.sourceId, job.entryId, propertyProvider); - if (allowCache && Config.cacheParams.useCache) { + if (allowCache && Config.cacheMaxSizeInBytes > 0) { StructureCache.add(ret); } return ret; diff --git a/src/servers/volume/config.ts b/src/servers/volume/config.ts index 7eb000cc857e6d4fadbfe8e968af16633036969c..4647394333be0bc905ac5a1fbfab83a1194b213f 100644 --- a/src/servers/volume/config.ts +++ b/src/servers/volume/config.ts @@ -1,14 +1,24 @@ /** - * Copyright (c) 2019 mol* contributors, licensed under MIT, See LICENSE file for more info. + * Copyright (c) 2019-2020 mol* contributors, licensed under MIT, See LICENSE file for more info. * * @author David Sehnal <david.sehnal@gmail.com> * @author Alexander Rose <alexander.rose@weirdbyte.de> */ import * as argparse from 'argparse' -import { ObjectKeys } from '../../mol-util/type-helpers'; +import { ObjectKeys } from '../../mol-util/type-helpers' +import { VOLUME_SERVER_HEADER, VOLUME_SERVER_VERSION } from './server/version' +import * as fs from 'fs' -export function addLimitsArgs(parser: argparse.ArgumentParser) { +const DefaultServerConfig = { + apiPrefix: '/VolumeServer', + defaultPort: 1337, + shutdownTimeoutMinutes: 24 * 60, /* a day */ + shutdownTimeoutVarianceMinutes: 60, + idMap: [] as [string, string][] +} + +function addLimitsArgs(parser: argparse.ArgumentParser) { parser.addArgument([ '--maxRequestBlockCount' ], { defaultValue: DefaultLimitsConfig.maxRequestBlockCount, metavar: 'COUNT', @@ -32,7 +42,7 @@ The maximum number of voxels is tied to maxRequestBlockCount.` }); } -export function addServerArgs(parser: argparse.ArgumentParser) { +function addServerArgs(parser: argparse.ArgumentParser) { parser.addArgument([ '--apiPrefix' ], { defaultValue: DefaultServerConfig.apiPrefix, metavar: 'PREFIX', @@ -65,15 +75,15 @@ export function addServerArgs(parser: argparse.ArgumentParser) { 'Map `id`s for a `type` to a file path.', 'Example: x-ray \'../../data/mdb/xray/${id}-ccp4.mdb\'', '', - ' - JS expressions can be used with in the ${}, e.g. \'${id.substr(1, 2)}/${id}.mdb\'', + ' - JS expressions can be used inside ${}, e.g. \'${id.substr(1, 2)}/${id}.mdb\'', ' - Can be specified multiple times.', - ' - The "TYPE" variable (e.g. "x-ray") is arbitrary and depends on how you plan to use the server.', - ' By default, Mol* Viewer uses "x-ray" and "em", but any particular use case may vary. ' + ' - The `TYPE` variable (e.g. `x-ray`) is arbitrary and depends on how you plan to use the server.', + ' By default, Mol* Viewer uses `x-ray` and `em`, but any particular use case may vary. ' ].join('\n'), }); } -export function addJsonConfigArgs(parser: argparse.ArgumentParser) { +function addJsonConfigArgs(parser: argparse.ArgumentParser) { parser.addArgument(['--cfg'], { help: [ 'JSON config file path', @@ -82,32 +92,25 @@ export function addJsonConfigArgs(parser: argparse.ArgumentParser) { required: false }); parser.addArgument(['--printCfg'], { help: 'Print current config for validation and exit.', required: false, nargs: 0 }); - parser.addArgument(['--printCfgTemplate'], { help: 'Prints default JSON config template to be modified and exits.', required: false, nargs: 0 }); + parser.addArgument(['--cfgTemplate'], { help: 'Prints default JSON config template to be modified and exits.', required: false, nargs: 0 }); } export interface ServerJsonConfig { cfg?: string, printCfg?: any, - printCfgTemplate?: any + cfgTemplate?: any } -const DefaultServerConfig = { - apiPrefix: '/VolumeServer', - defaultPort: 1337, - shutdownTimeoutMinutes: 24 * 60, /* a day */ - shutdownTimeoutVarianceMinutes: 60, - idMap: [] as [string, string][] -} export type ServerConfig = typeof DefaultServerConfig export const ServerConfig = { ...DefaultServerConfig } -export function setServerConfig(config: ServerConfig) { +function setServerConfig(config: ServerConfig) { for (const k of ObjectKeys(ServerConfig)) { - if (config[k]) (ServerConfig as any)[k] = config[k]; + if (config[k] !== void 0) (ServerConfig as any)[k] = config[k]; } } -export function validateServerConfig() { +function validateServerConfig() { if (!ServerConfig.idMap || ServerConfig.idMap.length === 0) { throw new Error(`Please provide 'idMap' configuration. See [-h] for available options.`); } @@ -128,18 +131,21 @@ const DefaultLimitsConfig = { } export type LimitsConfig = typeof DefaultLimitsConfig export const LimitsConfig = { ...DefaultLimitsConfig } -export function setLimitsConfig(config: LimitsConfig) { + +function setLimitsConfig(config: LimitsConfig) { for (const k of ObjectKeys(LimitsConfig)) { - if (config[k]) (LimitsConfig as any)[k] = config[k]; + if (config[k] !== void 0) (LimitsConfig as any)[k] = config[k]; } } -export function setConfig(config: ServerConfig & LimitsConfig) { +type FullServerConfig = ServerConfig & LimitsConfig + +function setConfig(config: FullServerConfig) { setServerConfig(config) setLimitsConfig(config) } -export const ServerConfigTemplate: ServerConfig & LimitsConfig = { +const ServerConfigTemplate: FullServerConfig = { ...DefaultServerConfig, idMap: [ ['x-ray', './path-to-xray-data/${id.substr(1, 2)}/${id}.mdb'], @@ -147,3 +153,77 @@ export const ServerConfigTemplate: ServerConfig & LimitsConfig = { ] as [string, string][], ...DefaultLimitsConfig } + +export function configureServer() { + const parser = new argparse.ArgumentParser({ + version: VOLUME_SERVER_VERSION, + addHelp: true, + description: VOLUME_SERVER_HEADER + }); + addJsonConfigArgs(parser); + addServerArgs(parser); + addLimitsArgs(parser); + const config = parser.parseArgs() as FullServerConfig & ServerJsonConfig; + + if (config.cfgTemplate !== null) { + console.log(JSON.stringify(ServerConfigTemplate, null, 2)); + process.exit(0); + } + + try { + setConfig(config) // sets the config for global use + + if (config.cfg) { + const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as FullServerConfig; + setConfig(cfg); + } + + if (config.printCfg !== null) { + console.log(JSON.stringify({ ...ServerConfig, ...LimitsConfig }, null, 2)); + process.exit(0); + } + + validateServerConfig(); + } catch (e) { + console.error('' + e); + process.exit(1); + } +} + +export function configureLocal() { + const parser = new argparse.ArgumentParser({ + version: VOLUME_SERVER_VERSION, + addHelp: true, + description: VOLUME_SERVER_HEADER + }); + parser.addArgument(['--jobs'], { help: `Path to a JSON file with job specification.`, required: false }); + parser.addArgument(['--jobsTemplate'], { help: 'Print example template for jobs.json and exit.', required: false, nargs: 0 });; + addJsonConfigArgs(parser); + addLimitsArgs(parser); + + const config = parser.parseArgs() as LimitsConfig & ServerJsonConfig; + + if (config.cfgTemplate !== null) { + console.log(JSON.stringify(DefaultLimitsConfig, null, 2)); + process.exit(0); + } + + try { + setLimitsConfig(config) // sets the config for global use + + if (config.cfg) { + const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as FullServerConfig; + setLimitsConfig(cfg); + } + + if (config.printCfg !== null) { + console.log(JSON.stringify(LimitsConfig, null, 2)); + process.exit(0); + } + + return config as LimitsConfig & { jobs: string, jobsTemplate: any }; + } catch (e) { + console.error('' + e); + process.exit(1); + } +} \ No newline at end of file diff --git a/src/servers/volume/local.ts b/src/servers/volume/query.ts similarity index 61% rename from src/servers/volume/local.ts rename to src/servers/volume/query.ts index f2145c2badfc82440a9a4ed0ed1c2cc68bc2aa76..85500172bf3712e9da74d339b4f04779d57adad6 100644 --- a/src/servers/volume/local.ts +++ b/src/servers/volume/query.ts @@ -7,16 +7,13 @@ * @author Alexander Rose <alexander.rose@weirdbyte.de> */ -import * as argparse from 'argparse' -import * as LocalApi from './server/local-api' -import VERSION from './server/version' -import * as fs from 'fs' -import { LimitsConfig, addLimitsArgs, setLimitsConfig } from './config'; +import * as fs from 'fs'; +import { configureLocal } from './config'; +import * as LocalApi from './server/local-api'; -console.log(`VolumeServer Local ${VERSION}, (c) 2018-2019, Mol* contributors`); -console.log(); +const config = configureLocal(); -function description() { +if (config.jobsTemplate !== null) { const exampleJobs: LocalApi.JobEntry[] = [{ source: { filename: `g:/test/mdb/xray-1tqn.mdb`, @@ -49,29 +46,20 @@ function description() { }, outputFolder: 'g:/test/local-test' }]; - - return `Usage: node local jobs.json\n\nExample jobs.json: ${JSON.stringify(exampleJobs, null, 2)}` + console.log(JSON.stringify(exampleJobs, null, 2)); + process.exit(); } -const parser = new argparse.ArgumentParser({ - addHelp: true, - description: description() -}); -addLimitsArgs(parser) -parser.addArgument(['jobs'], { - help: `Path to jobs JSON file.` -}) - -const config: LimitsConfig & { jobs: string } = parser.parseArgs() -setLimitsConfig(config) // sets the config for global use - async function run() { let jobs: LocalApi.JobEntry[]; try { + if (!config.jobs) { + throw new Error(`Please provide 'jobs' argument. See [-h] for help.`); + } + jobs = JSON.parse(fs.readFileSync(config.jobs, 'utf-8')); } catch (e) { - console.log('Error:'); - console.error(e); + console.error('' + e); return; } diff --git a/src/servers/volume/server.ts b/src/servers/volume/server.ts index 9d74dabb599cd3dd1b00c790cb892716c72d24fa..c4c2b02710d65776742e4270021274d5205e0e81 100644 --- a/src/servers/volume/server.ts +++ b/src/servers/volume/server.ts @@ -7,16 +7,14 @@ * @author Alexander Rose <alexander.rose@weirdbyte.de> */ -import * as express from 'express' import * as compression from 'compression' - -import init from './server/web-api' -import VERSION from './server/version' +import * as express from 'express' import { ConsoleLogger } from '../../mol-util/console-logger' +import { configureServer, ServerConfig } from './config' import { State } from './server/state' -import { addServerArgs, addLimitsArgs, LimitsConfig, setConfig, ServerConfig, addJsonConfigArgs, ServerJsonConfig, ServerConfigTemplate, validateServerConfig } from './config'; -import * as argparse from 'argparse' -import * as fs from 'fs' +import { VOLUME_SERVER_HEADER } from './server/version' +import init from './server/web-api' + function setupShutdown() { if (ServerConfig.shutdownTimeoutVarianceMinutes > ServerConfig.shutdownTimeoutMinutes) { @@ -45,39 +43,7 @@ function setupShutdown() { } } -const parser = new argparse.ArgumentParser({ - addHelp: true, - description: `VolumeServer ${VERSION}, (c) 2018-2019, Mol* contributors` -}); -addJsonConfigArgs(parser); -addServerArgs(parser) -addLimitsArgs(parser) - -const config: ServerConfig & LimitsConfig & ServerJsonConfig = parser.parseArgs() - -if (config.printCfgTemplate !== null) { - console.log(JSON.stringify(ServerConfigTemplate, null, 2)); - process.exit(0); -} - -try { - setConfig(config) // sets the config for global use - - if (config.cfg) { - const cfg = JSON.parse(fs.readFileSync(config.cfg, 'utf8')) as ServerConfig & LimitsConfig; - setConfig(cfg); - } - - if (config.printCfg !== null) { - console.log(JSON.stringify({ ...ServerConfig, ...LimitsConfig }, null, 2)); - process.exit(0); - } - - validateServerConfig(); -} catch (e) { - console.error('' + e); - process.exit(1); -} +configureServer(); const port = process.env.port || ServerConfig.defaultPort; @@ -87,11 +53,11 @@ init(app); app.listen(port); -console.log(`VolumeServer ${VERSION}, (c) 2018-2019, Mol* contributors`); +console.log(VOLUME_SERVER_HEADER); console.log(``); console.log(`The server is running on port ${port}.`); console.log(``); -if (config.shutdownTimeoutMinutes > 0) { +if (ServerConfig.shutdownTimeoutMinutes > 0) { setupShutdown(); } \ No newline at end of file diff --git a/src/servers/volume/server/query/encode.ts b/src/servers/volume/server/query/encode.ts index 6efdfc34c04f1da4686d14126cab6af4c8a915a5..b89691f180ea4e1f46f687d0010f2ac571c0f28a 100644 --- a/src/servers/volume/server/query/encode.ts +++ b/src/servers/volume/server/query/encode.ts @@ -9,7 +9,7 @@ import { CifWriter } from '../../../../mol-io/writer/cif' import * as Data from './data-model' import * as Coords from '../algebra/coordinate' -import VERSION from '../version' +import { VOLUME_SERVER_VERSION as VERSION } from '../version' import * as DataFormat from '../../common/data-format' import { Column } from '../../../../mol-data/db'; import { ArrayEncoding, ArrayEncoder } from '../../../../mol-io/common/binary-cif'; diff --git a/src/servers/volume/server/version.ts b/src/servers/volume/server/version.ts index bf75cf10f9d549016d6c8d06e4aab63cb28098db..806aa778ef648eb7ea2212600271d4247d2649fd 100644 --- a/src/servers/volume/server/version.ts +++ b/src/servers/volume/server/version.ts @@ -1 +1,2 @@ -export default '0.9.5' \ No newline at end of file +export const VOLUME_SERVER_VERSION = '0.9.5' +export const VOLUME_SERVER_HEADER = `VolumeServer ${VOLUME_SERVER_VERSION}, (c) 2018-2020, Mol* contributors` \ No newline at end of file diff --git a/src/servers/volume/server/web-schema.ts b/src/servers/volume/server/web-schema.ts index fe7bc7db8d229a6133712853be14e2f53739dd33..e9a1d4684b1380d809eb82092fda45ffdd50674b 100644 --- a/src/servers/volume/server/web-schema.ts +++ b/src/servers/volume/server/web-schema.ts @@ -5,7 +5,7 @@ * @author David Sehnal <david.sehnal@gmail.com> */ -import VERSION from './version' +import { VOLUME_SERVER_VERSION } from './version' import { LimitsConfig, ServerConfig } from '../config'; export function getSchema() { @@ -18,7 +18,7 @@ export function getSchema() { return { openapi: '3.0.0', info: { - version: VERSION, + version: VOLUME_SERVER_VERSION, title: 'Volume Server', description: 'The VolumeServer is a service for accessing subsets of volumetric data. It automatically downsamples the data depending on the volume of the requested region to reduce the bandwidth requirements and provide near-instant access to even the largest data sets.', },