Merge branch 'develop' into pg-support

- for testing forwarded ip address later

.github/ISSUE_TEMPLATE/bug_report.md

@@ -6,15 +6,30 @@ labels: bug
 assignees: ''
 
 ---
+<!--
+ 
+Are you in the right place?
+- If you are looking for support on how to get your upstream server forwarding, please consider asking the community on Reddit.
+- If you are writing code changes to contribute and need to ask about the internals of the software, Gitter is the best place to ask.
+- If you think you found a bug with NPM (not Nginx, or your upstream server or MySql) then you are in the *right place.*
+
+-->
 
 **Checklist**
 - Have you pulled and found the error with `jc21/nginx-proxy-manager:latest` docker image?
+  - Yes / No
 - Are you sure you're not using someone else's docker image?
-- If having problems with Lets Encrypt, have you made absolutely sure your site is accessible from outside of your network?
+  - Yes / No
+- Have you searched for similar issues (both open and closed)?
+  - Yes / No
 
 **Describe the bug**
-- A clear and concise description of what the bug is.
+<!-- A clear and concise description of what the bug is. -->
-- What version of Nginx Proxy Manager is reported on the login page?
+
+
+**Nginx Proxy Manager Version**
+<!-- What version of Nginx Proxy Manager is reported on the login page? -->
+
 
 **To Reproduce**
 Steps to reproduce the behavior:
@@ -23,14 +38,18 @@ Steps to reproduce the behavior:
 3. Scroll down to '....'
 4. See error
 
+
 **Expected behavior**
-A clear and concise description of what you expected to happen.
+<!-- A clear and concise description of what you expected to happen. -->
+
 
 **Screenshots**
-If applicable, add screenshots to help explain your problem.
+<!-- If applicable, add screenshots to help explain your problem. -->
+
 
 **Operating System**
-- Please specify if using a Rpi, Mac, orchestration tool or any other setups that might affect the reproduction of this error.
+<!-- Please specify if using a Rpi, Mac, orchestration tool or any other setups that might affect the reproduction of this error. -->
+
 
 **Additional context**
-Add any other context about the problem here, docker version, browser version if applicable to the problem. Too much info is better than too little.
+<!-- Add any other context about the problem here, docker version, browser version, logs if applicable to the problem. Too much info is better than too little. -->

.github/ISSUE_TEMPLATE/dns_challenge_request.md

@@ -0,0 +1,18 @@
+---
+name: DNS challenge provider request
+about: Suggest a new provider to be available for a certificate DNS challenge
+title: ''
+labels: dns provider request
+assignees: ''
+
+---
+
+**What provider would you like to see added to NPM?**
+<!-- What is this provider called? -->
+
+
+**Have you checked if a certbot plugin exists?**
+<!-- 
+Currently NPM only supports DNS challenge providers for which a certbot plugin exists. 
+You can visit pypi.org, and search for a package with the name `certbot-dns-<privider>`.
+-->

.github/ISSUE_TEMPLATE/feature_request.md

@@ -7,14 +7,26 @@ assignees: ''
 
 ---
 
+<!--
+
+Are you in the right place?
+- If you are looking for support on how to get your upstream server forwarding, please consider asking the community on Reddit.
+- If you are writing code changes to contribute and need to ask about the internals of the software, Gitter is the best place to ask.
+- If you think you found a bug with NPM (not Nginx, or your upstream server or MySql) then you are in the *right place.*
+
+-->
+
 **Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
+
 
 **Describe the solution you'd like**
-A clear and concise description of what you want to happen.
+<!-- A clear and concise description of what you want to happen. -->
+
 
 **Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
+
 
 **Additional context**
-Add any other context or screenshots about the feature request here.
+<!-- Add any other context or screenshots about the feature request here. -->

.github/workflows/stale.yml

@@ -0,0 +1,21 @@
+name: 'Close stale issues and PRs'
+on:
+  schedule:
+    - cron: '30 1 * * *'
+  workflow_dispatch:
+
+jobs:
+  stale:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@v9
+        with:
+          stale-issue-label: 'stale'
+          stale-pr-label: 'stale'
+          stale-issue-message: 'Issue is now considered stale. If you want to keep it open, please comment :+1:'
+          stale-pr-message: 'PR is now considered stale. If you want to keep it open, please comment :+1:'
+          close-issue-message: 'Issue was closed due to inactivity.'
+          close-pr-message: 'PR was closed due to inactivity.'
+          days-before-stale: 182
+          days-before-close: 365
+          operations-per-run: 50

.gitignore

@@ -2,4 +2,8 @@
 .idea
 ._*
 .vscode
-
+certbot-help.txt
+test/node_modules
+*/node_modules
+docker/dev/dnsrouter-config.json.tmp
+docker/dev/resolv.conf

.jenkins/config.json

@@ -1,10 +0,0 @@
-{
-	"database": {
-		"engine": "mysql",
-		"host": "db",
-		"name": "npm",
-		"user": "npm",
-		"password": "npm",
-		"port": 3306
-	}
-}

.version

@@ -1 +1 @@
-2.2.1
+2.12.0

Jenkinsfile

@@ -1,3 +1,9 @@
+import groovy.transform.Field
+
+@Field
+def shOutput = ""
+def buildxPushTags = ""
+
 pipeline {
 	agent {
 		label 'docker-multiarch'
@@ -5,16 +11,15 @@ pipeline {
 	options {
 		buildDiscarder(logRotator(numToKeepStr: '5'))
 		disableConcurrentBuilds()
+		ansiColor('xterm')
 	}
 	environment {
-		IMAGE                      = "nginx-proxy-manager"
+		IMAGE                      = 'nginx-proxy-manager'
 		BUILD_VERSION              = getVersion()
-		MAJOR_VERSION              = "2"
+		MAJOR_VERSION              = '2'
-		BRANCH_LOWER               = "${BRANCH_NAME.toLowerCase().replaceAll('/', '-')}"
+		BRANCH_LOWER               = "${BRANCH_NAME.toLowerCase().replaceAll('\\\\', '-').replaceAll('/', '-').replaceAll('\\.', '-')}"
-		COMPOSE_PROJECT_NAME       = "npm_${BRANCH_LOWER}_${BUILD_NUMBER}"
+		BUILDX_NAME                = "npm_${BRANCH_LOWER}_${BUILD_NUMBER}"
-		COMPOSE_FILE               = 'docker/docker-compose.ci.yml'
 		COMPOSE_INTERACTIVE_NO_CLI = 1
-		BUILDX_NAME                = "${COMPOSE_PROJECT_NAME}"
 	}
 	stages {
 		stage('Environment') {
@@ -25,7 +30,7 @@ pipeline {
 					}
 					steps {
 						script {
-							env.BUILDX_PUSH_TAGS = "-t docker.io/jc21/${IMAGE}:${BUILD_VERSION} -t docker.io/jc21/${IMAGE}:${MAJOR_VERSION} -t docker.io/jc21/${IMAGE}:latest"
+							buildxPushTags = "-t docker.io/jc21/${IMAGE}:${BUILD_VERSION} -t docker.io/jc21/${IMAGE}:${MAJOR_VERSION} -t docker.io/jc21/${IMAGE}:latest"
 						}
 					}
 				}
@@ -38,7 +43,7 @@ pipeline {
 					steps {
 						script {
 							// Defaults to the Branch name, which is applies to all branches AND pr's
-							env.BUILDX_PUSH_TAGS = "-t docker.io/jc21/${IMAGE}:github-${BRANCH_LOWER}"
+							buildxPushTags = "-t docker.io/jc21/${IMAGE}:github-${BRANCH_LOWER}"
 						}
 					}
 				}
@@ -51,87 +56,104 @@ pipeline {
 						sh 'sed -i -E "s/(version-)[0-9]+\\.[0-9]+\\.[0-9]+(-green)/\\1${BUILD_VERSION}\\2/" README.md'
 					}
 				}
-			}
+				stage('Docker Login') {
-		}
-		stage('Frontend') {
 					steps {
-				ansiColor('xterm') {
+						withCredentials([usernamePassword(credentialsId: 'jc21-dockerhub', passwordVariable: 'dpass', usernameVariable: 'duser')]) {
-					sh './scripts/frontend-build'
+							sh 'docker login -u "${duser}" -p "${dpass}"'
 						}
 					}
 				}
-		stage('Backend') {
+			}
+		}
+		stage('Builds') {
+			parallel {
+				stage('Project') {
 					steps {
-				ansiColor('xterm') {
+						script {
-					echo 'Checking Syntax ...'
+							// Frontend and Backend
-					// See: https://github.com/yarnpkg/yarn/issues/3254
+							def shStatusCode = sh(label: 'Checking and Building', returnStatus: true, script: '''
-					sh '''docker run --rm \\
+								set -e
-						-v "$(pwd)/backend:/app" \\
+								./scripts/ci/frontend-build > ${WORKSPACE}/tmp-sh-build 2>&1
-						-w /app \\
+								./scripts/ci/test-and-build > ${WORKSPACE}/tmp-sh-build 2>&1
-						node:latest \\
+							''')
-						sh -c "yarn install && yarn eslint . && rm -rf node_modules"
+							shOutput = readFile "${env.WORKSPACE}/tmp-sh-build"
-					'''
+							if (shStatusCode != 0) {
-
+								error "${shOutput}"
-					echo 'Docker Build ...'
-					sh '''docker build --pull --no-cache --squash --compress \\
-						-t "${IMAGE}:ci-${BUILD_NUMBER}" \\
-						-f docker/Dockerfile \\
-						--build-arg TARGETPLATFORM=linux/amd64 \\
-						--build-arg BUILDPLATFORM=linux/amd64 \\
-						--build-arg BUILD_VERSION="${BUILD_VERSION}" \\
-						--build-arg BUILD_COMMIT="${BUILD_COMMIT}" \\
-						--build-arg BUILD_DATE="$(date '+%Y-%m-%d %T %Z')" \\
-						.
-					'''
 							}
 						}
 					}
-		stage('Test') {
-			steps {
-				ansiColor('xterm') {
-					// Bring up a stack
-					sh 'docker-compose up -d fullstack'
-					sh './scripts/wait-healthy $(docker-compose ps -q fullstack) 120'
-
-					// Run tests
-					sh 'rm -rf test/results'
-					sh 'docker-compose up cypress'
-					// Get results
-					sh 'docker cp -L "$(docker-compose ps -q cypress):/results" test/'
-				}
-			}
 					post {
 						always {
-					// Dumps to analyze later
+							sh 'rm -f ${WORKSPACE}/tmp-sh-build'
-					sh 'mkdir -p debug'
-					sh 'docker-compose logs fullstack | gzip > debug/docker_fullstack.log.gz'
-					sh 'docker-compose logs db | gzip > debug/docker_db.log.gz'
-					// Cypress videos and screenshot artifacts
-					dir(path: 'test/results') {
-						archiveArtifacts allowEmptyArchive: true, artifacts: '**/*', excludes: '**/*.xml'
 						}
-					junit 'test/results/junit/*'
+						failure {
+							npmGithubPrComment("CI Error:\n\n```\n${shOutput}\n```", true)
 						}
 					}
 				}
 				stage('Docs') {
+					steps {
+						dir(path: 'docs') {
+							sh 'yarn install'
+							sh 'yarn build'
+						}
+					}
+				}
+			}
+		}
+		stage('Test Sqlite') {
+			environment {
+				COMPOSE_PROJECT_NAME = "npm_${BRANCH_LOWER}_${BUILD_NUMBER}_sqlite"
+				COMPOSE_FILE         = 'docker/docker-compose.ci.yml:docker/docker-compose.ci.sqlite.yml'
+			}
 			when {
 				not {
 					equals expected: 'UNSTABLE', actual: currentBuild.result
 				}
 			}
 			steps {
-				ansiColor('xterm') {
+				sh 'rm -rf ./test/results/junit/*'
-					dir(path: 'docs') {
+				sh './scripts/ci/fulltest-cypress'
-						sh 'yarn install'
-						sh 'yarn build'
 			}
-
+			post {
-					dir(path: 'docs/.vuepress/dist') {
+				always {
-						sh 'tar -czf ../../docs.tgz *'
+					// Dumps to analyze later
+					sh 'mkdir -p debug/sqlite'
+					sh 'docker logs $(docker-compose ps --all -q fullstack) > debug/sqlite/docker_fullstack.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q stepca) > debug/sqlite/docker_stepca.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q pdns) > debug/sqlite/docker_pdns.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q pdns-db) > debug/sqlite/docker_pdns-db.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q dnsrouter) > debug/sqlite/docker_dnsrouter.log 2>&1'
+					junit 'test/results/junit/*'
+					sh 'docker-compose down --remove-orphans --volumes -t 30 || true'
 				}
-
+			}
-					archiveArtifacts(artifacts: 'docs/docs.tgz', allowEmptyArchive: false)
+		}
+		stage('Test Mysql') {
+			environment {
+				COMPOSE_PROJECT_NAME = "npm_${BRANCH_LOWER}_${BUILD_NUMBER}_mysql"
+				COMPOSE_FILE         = 'docker/docker-compose.ci.yml:docker/docker-compose.ci.mysql.yml'
+			}
+			when {
+				not {
+					equals expected: 'UNSTABLE', actual: currentBuild.result
+				}
+			}
+			steps {
+				sh 'rm -rf ./test/results/junit/*'
+				sh './scripts/ci/fulltest-cypress'
+			}
+			post {
+				always {
+					// Dumps to analyze later
+					sh 'mkdir -p debug/mysql'
+					sh 'docker logs $(docker-compose ps --all -q fullstack) > debug/mysql/docker_fullstack.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q stepca) > debug/mysql/docker_stepca.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q pdns) > debug/mysql/docker_pdns.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q pdns-db) > debug/mysql/docker_pdns-db.log 2>&1'
+					sh 'docker logs $(docker-compose ps --all -q dnsrouter) > debug/mysql/docker_dnsrouter.log 2>&1'
+					junit 'test/results/junit/*'
+					sh 'docker-compose down --remove-orphans --volumes -t 30 || true'
 				}
 			}
 		}
@@ -142,45 +164,22 @@ pipeline {
 				}
 			}
 			steps {
-				ansiColor('xterm') {
+				sh "./scripts/buildx --push ${buildxPushTags}"
-					withCredentials([usernamePassword(credentialsId: 'jc21-dockerhub', passwordVariable: 'dpass', usernameVariable: 'duser')]) {
-						sh "docker login -u '${duser}' -p '${dpass}'"
-						// Buildx with push
-						sh "./scripts/buildx --push ${BUILDX_PUSH_TAGS}"
 			}
 		}
-			}
+		stage('Docs / Comment') {
-		}
+			parallel {
-		stage('Docs Deploy') {
+				stage('Docs Job') {
 					when {
 						allOf {
-					branch 'master'
+							branch pattern: "^(develop|master)\$", comparator: "REGEXP"
 							not {
 								equals expected: 'UNSTABLE', actual: currentBuild.result
 							}
 						}
 					}
 					steps {
-				withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', accessKeyVariable: 'AWS_ACCESS_KEY_ID', credentialsId: 'npm-s3-docs', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
+						build wait: false, job: 'nginx-proxy-manager-docs', parameters: [string(name: 'docs_branch', value: "$BRANCH_NAME")]
-					sh """docker run --rm \\
-						--name \${COMPOSE_PROJECT_NAME}-docs-upload \\
-						-e S3_BUCKET=jc21-npm-site \\
-						-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \\
-						-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \\
-						-v \$(pwd):/app \\
-						-w /app \\
-						jc21/ci-tools \\
-						scripts/docs-upload /app/docs/.vuepress/dist/
-					"""
-
-					sh """docker run --rm \\
-						--name \${COMPOSE_PROJECT_NAME}-docs-invalidate \\
-						-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \\
-						-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \\
-						jc21/ci-tools \\
-						aws cloudfront create-invalidation --distribution-id EN1G6DEWZUTDT --paths '/*'
-					"""
-				}
 					}
 				}
 				stage('PR Comment') {
@@ -193,9 +192,9 @@ pipeline {
 						}
 					}
 					steps {
-				ansiColor('xterm') {
 						script {
-						def comment = pullRequest.comment("Docker Image for build ${BUILD_NUMBER} is available on [DockerHub](https://cloud.docker.com/repository/docker/jc21/${IMAGE}) as `jc21/${IMAGE}:github-${BRANCH_LOWER}`")
+							npmGithubPrComment("Docker Image for build ${BUILD_NUMBER} is available on [DockerHub](https://cloud.docker.com/repository/docker/jc21/${IMAGE}) as `jc21/${IMAGE}:github-${BRANCH_LOWER}`\n\n**Note:** ensure you backup your NPM instance before testing this PR image! Especially if this PR contains database changes.", true)
+						}
 					}
 				}
 			}
@@ -203,21 +202,20 @@ pipeline {
 	}
 	post {
 		always {
-			sh 'docker-compose down --rmi all --remove-orphans --volumes -t 30'
 			sh 'echo Reverting ownership'
-			sh 'docker run --rm -v $(pwd):/data ${DOCKER_CI_TOOLS} chown -R $(id -u):$(id -g) /data'
+			sh 'docker run --rm -v "$(pwd):/data" jc21/ci-tools chown -R "$(id -u):$(id -g)" /data'
 		}
 		success {
 			juxtapose event: 'success'
 			sh 'figlet "SUCCESS"'
 		}
 		failure {
-			archiveArtifacts(artifacts: 'debug/**.*', allowEmptyArchive: true)
+			archiveArtifacts(artifacts: 'debug/**/*.*', allowEmptyArchive: true)
 			juxtapose event: 'failure'
 			sh 'figlet "FAILURE"'
 		}
 		unstable {
-			archiveArtifacts(artifacts: 'debug/**.*', allowEmptyArchive: true)
+			archiveArtifacts(artifacts: 'debug/**/*.*', allowEmptyArchive: true)
 			juxtapose event: 'unstable'
 			sh 'figlet "UNSTABLE"'
 		}

README.md

@@ -1,28 +1,25 @@
 <p align="center">
 	<img src="https://nginxproxymanager.com/github.png">
 	<br><br>
-  <img src="https://img.shields.io/badge/version-2.2.1-green.svg?style=for-the-badge">
+	<img src="https://img.shields.io/badge/version-2.12.0-green.svg?style=for-the-badge">
 	<a href="https://hub.docker.com/repository/docker/jc21/nginx-proxy-manager">
 		<img src="https://img.shields.io/docker/stars/jc21/nginx-proxy-manager.svg?style=for-the-badge">
 	</a>
 	<a href="https://hub.docker.com/repository/docker/jc21/nginx-proxy-manager">
 		<img src="https://img.shields.io/docker/pulls/jc21/nginx-proxy-manager.svg?style=for-the-badge">
 	</a>
-  <a href="https://ci.nginxproxymanager.com/blue/organizations/jenkins/nginx-proxy-manager/branches/">
-    <img src="https://img.shields.io/jenkins/build?jobUrl=https%3A%2F%2Fci.nginxproxymanager.com%2Fjob%2Fnginx-proxy-manager%2Fjob%2Fmaster&style=for-the-badge">
-  </a>
 </p>
 
 This project comes as a pre-built docker image that enables you to easily forward to your websites
 running at home or otherwise, including free SSL, without having to know too much about Nginx or Letsencrypt.
 
-- [Quick Setup](https://nginxproxymanager.com#quick-setup)
+- [Quick Setup](#quick-setup)
 - [Full Setup](https://nginxproxymanager.com/setup/)
 - [Screenshots](https://nginxproxymanager.com/screenshots/)
 
 ## Project Goal
 
-I created this project to fill a personal need to provide users with a easy way to accomplish reverse
+I created this project to fill a personal need to provide users with an easy way to accomplish reverse
 proxying hosts with SSL termination and it had to be so easy that a monkey could do it. This goal hasn't changed.
 While there might be advanced options they are optional and the project should be as simple as possible
 so that the barrier for entry here is low.
@@ -48,3 +45,76 @@ I won't go in to too much detail here but here are the basics for someone new to
 2. Add port forwarding for port 80 and 443 to the server hosting this project
 3. Configure your domain name details to point to your home, either with a static ip or a service like DuckDNS or [Amazon Route53](https://github.com/jc21/route53-ddns)
 4. Use the Nginx Proxy Manager as your gateway to forward to your other web based services
+
+## Quick Setup
+
+1. Install Docker and Docker-Compose
+
+- [Docker Install documentation](https://docs.docker.com/install/)
+- [Docker-Compose Install documentation](https://docs.docker.com/compose/install/)
+
+2. Create a docker-compose.yml file similar to this:
+
+```yml
+services:
+  app:
+    image: 'docker.io/jc21/nginx-proxy-manager:latest'
+    restart: unless-stopped
+    ports:
+      - '80:80'
+      - '81:81'
+      - '443:443'
+    volumes:
+      - ./data:/data
+      - ./letsencrypt:/etc/letsencrypt
+```
+
+This is the bare minimum configuration required. See the [documentation](https://nginxproxymanager.com/setup/) for more.
+
+3. Bring up your stack by running
+
+```bash
+docker-compose up -d
+
+# If using docker-compose-plugin
+docker compose up -d
+
+```
+
+4. Log in to the Admin UI
+
+When your docker container is running, connect to it on port `81` for the admin interface.
+Sometimes this can take a little bit because of the entropy of keys.
+
+[http://127.0.0.1:81](http://127.0.0.1:81)
+
+Default Admin User:
+```
+Email:    admin@example.com
+Password: changeme
+```
+
+Immediately after logging in with this default user you will be asked to modify your details and change your password.
+
+
+## Contributing
+
+All are welcome to create pull requests for this project, against the `develop` branch. Official releases are created from the `master` branch.
+
+CI is used in this project. All PR's must pass before being considered. After passing,
+docker builds for PR's are available on dockerhub for manual verifications.
+
+Documentation within the `develop` branch is available for preview at
+[https://develop.nginxproxymanager.com](https://develop.nginxproxymanager.com)
+
+
+### Contributors
+
+Special thanks to [all of our contributors](https://github.com/NginxProxyManager/nginx-proxy-manager/graphs/contributors).
+
+
+## Getting Support
+
+1. [Found a bug?](https://github.com/NginxProxyManager/nginx-proxy-manager/issues)
+2. [Discussions](https://github.com/NginxProxyManager/nginx-proxy-manager/discussions)
+3. [Reddit](https://reddit.com/r/nginxproxymanager)

backend/.gitignore

@@ -4,3 +4,5 @@ yarn-error.log
 tmp
 certbot.log
 node_modules
+core.*
+

backend/.vscode/settings.json

@@ -1,8 +0,0 @@
-{
-	"editor.insertSpaces": false,
-	"editor.formatOnSave": true,
-	"files.trimTrailingWhitespace": true,
-	"editor.codeActionsOnSave": {
-		"source.fixAll.eslint": true
-	}
-}

backend/app.js

@@ -2,6 +2,7 @@ const express     = require('express');
 const bodyParser  = require('body-parser');
 const fileUpload  = require('express-fileupload');
 const compression = require('compression');
+const config      = require('./lib/config');
 const log         = require('./logger').express;
 
 /**
@@ -24,7 +25,7 @@ app.enable('trust proxy', ['loopback', 'linklocal', 'uniquelocal']);
 app.enable('strict routing');
 
 // pretty print JSON when not live
-if (process.env.NODE_ENV !== 'production') {
+if (config.debug()) {
 	app.set('json spaces', 2);
 }
 
@@ -40,7 +41,6 @@ app.use(function (req, res, next) {
 	}
 
 	res.set({
-		'Strict-Transport-Security': 'includeSubDomains; max-age=631138519; preload',
 		'X-XSS-Protection':       '1; mode=block',
 		'X-Content-Type-Options': 'nosniff',
 		'X-Frame-Options':        x_frame_options,
@@ -52,7 +52,7 @@ app.use(function (req, res, next) {
 });
 
 app.use(require('./lib/express/jwt')());
-app.use('/', require('./routes/api/main'));
+app.use('/', require('./routes/main'));
 
 // production error handler
 // no stacktraces leaked to user
@@ -66,7 +66,7 @@ app.use(function (err, req, res, next) {
 		}
 	};
 
-	if (process.env.NODE_ENV === 'development') {
+	if (config.debug() || (req.baseUrl + req.path).includes('nginx/certificates')) {
 		payload.debug = {
 			stack:    typeof err.stack !== 'undefined' && err.stack ? err.stack.split('\n') : null,
 			previous: err.previous
@@ -75,7 +75,7 @@ app.use(function (err, req, res, next) {
 
 	// Not every error is worth logging - but this is good for now until it gets annoying.
 	if (typeof err.stack !== 'undefined' && err.stack) {
-		if (process.env.NODE_ENV === 'development') {
+		if (config.debug()) {
 			log.debug(err.stack);
 		} else if (typeof err.public == 'undefined' || !err.public) {
 			log.warn(err.message);

backend/config/default.json

@@ -1,6 +1,6 @@
 {
   "database": {
-    "engine": "mysql",
+    "engine": "mysql2",
     "host": "db",
     "name": "npm",
     "user": "npm",

backend/config/sqlite-test-db.json

@@ -0,0 +1,26 @@
+{
+  "database": {
+      "engine": "knex-native",
+      "knex": {
+        "client": "sqlite3",
+        "connection": {
+          "filename": "/app/config/mydb.sqlite"
+        },
+        "pool": {
+          "min": 0,
+          "max": 1,
+          "createTimeoutMillis": 3000,
+          "acquireTimeoutMillis": 30000,
+          "idleTimeoutMillis": 30000,
+          "reapIntervalMillis": 1000,
+          "createRetryIntervalMillis": 100,
+          "propagateCreateError": false
+        },
+        "migrations": {
+          "tableName": "migrations",
+          "stub": "src/backend/lib/migrate_template.js",
+          "directory": "src/backend/migrations"
+        }
+      }
+    }
+}

backend/db.js

@@ -1,25 +1,43 @@
-const config = require('config');
+const config = require('./lib/config');
 
 if (!config.has('database')) {
-	throw new Error('Database config does not exist! Please read the instructions: https://github.com/jc21/nginx-proxy-manager/blob/master/doc/INSTALL.md');
+	throw new Error('Database config does not exist! Please read the instructions: https://nginxproxymanager.com/setup/');
 }
 
-let data = {
+function generateDbConfig() {
-	client:     config.database.engine,
+	const cfg = config.get('database');
+	if (cfg.engine === 'knex-native') {
+		return cfg.knex;
+	}
+	if (cfg.engine === 'pg') {
+
+		return {
+			client:     cfg.engine,
 			connection: {
-		host:     config.database.host,
+				host:     cfg.host,
-		user:     config.database.user,
+				user:     cfg.user,
-		password: config.database.password,
+				password: cfg.password,
-		database: config.database.name,
+				database: cfg.name,
-		port:     config.database.port
+				port:     cfg.port
 			},
 			migrations: {
 				tableName: 'migrations'
 			}
-};
+		};
-
+	}
-if (typeof config.database.version !== 'undefined') {
+	return {
-	data.version = config.database.version;
+		client:     cfg.engine,
+		connection: {
+			host:     cfg.host,
+			user:     cfg.user,
+			password: cfg.password,
+			database: cfg.name,
+			port:     cfg.port
+		},
+		migrations: {
+			tableName: 'migrations'
+		}
+	};
 }
 
-module.exports = require('knex')(data);
+module.exports = require('knex')(generateDbConfig());

backend/index.js

@@ -1,23 +1,20 @@
 #!/usr/bin/env node
 
+const schema = require('./schema');
 const logger = require('./logger').global;
 
-function appStart () {
+async function appStart () {
 	const migrate             = require('./migrate');
 	const setup               = require('./setup');
 	const app                 = require('./app');
-	const apiValidator        = require('./lib/validator/api');
 	const internalCertificate = require('./internal/certificate');
 	const internalIpRanges    = require('./internal/ip_ranges');
 
 	return migrate.latest()
 		.then(setup)
-		.then(() => {
+		.then(schema.getCompiledSchema)
-			return apiValidator.loadSchemas;
-		})
 		.then(internalIpRanges.fetch)
 		.then(() => {
-
 			internalCertificate.initTimer();
 			internalIpRanges.initTimer();
 
@@ -34,7 +31,7 @@ function appStart () {
 			});
 		})
 		.catch((err) => {
-			logger.error(err.message);
+			logger.error(err.message, err);
 			setTimeout(appStart, 1000);
 		});
 }
@@ -45,3 +42,4 @@ try {
 	logger.error(err.message, err);
 	process.exit(1);
 }
+

backend/internal/access-list.js

@@ -3,12 +3,13 @@ const fs                  = require('fs');
 const batchflow             = require('batchflow');
 const logger                = require('../logger').access;
 const error                 = require('../lib/error');
+const utils                 = require('../lib/utils');
 const accessListModel       = require('../models/access_list');
 const accessListAuthModel   = require('../models/access_list_auth');
+const accessListClientModel = require('../models/access_list_client');
 const proxyHostModel        = require('../models/proxy_host');
 const internalAuditLog      = require('./audit-log');
 const internalNginx         = require('./nginx');
-const utils               = require('../lib/utils');
 
 function omissions () {
 	return ['is_deleted'];
@@ -26,17 +27,20 @@ const internalAccessList = {
 			.then((/*access_data*/) => {
 				return accessListModel
 					.query()
-					.omit(omissions())
 					.insertAndFetch({
 						name:          data.name,
+						satisfy_any:   data.satisfy_any,
+						pass_auth:     data.pass_auth,
 						owner_user_id: access.token.getUserId(1)
-					});
+					})
+					.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
 				data.id = row.id;
 
-				// Now add the items
 				let promises = [];
+
+				// Now add the items
 				data.items.map((item) => {
 					promises.push(accessListAuthModel
 						.query()
@@ -48,13 +52,27 @@ const internalAccessList = {
 					);
 				});
 
+				// Now add the clients
+				if (typeof data.clients !== 'undefined' && data.clients) {
+					data.clients.map((client) => {
+						promises.push(accessListClientModel
+							.query()
+							.insert({
+								access_list_id: row.id,
+								address:        client.address,
+								directive:      client.directive
+							})
+						);
+					});
+				}
+
 				return Promise.all(promises);
 			})
 			.then(() => {
 				// re-fetch with expansions
 				return internalAccessList.get(access, {
 					id:     data.id,
-					expand: ['owner', 'items']
+					expand: ['owner', 'items', 'clients', 'proxy_hosts.access_list.[clients,items]']
 				}, true /* <- skip masking */);
 			})
 			.then((row) => {
@@ -64,7 +82,7 @@ const internalAccessList = {
 				return internalAccessList.build(row)
 					.then(() => {
 						if (row.proxy_host_count) {
-							return internalNginx.reload();
+							return internalNginx.bulkGenerateConfigs('proxy_host', row.proxy_hosts);
 						}
 					})
 					.then(() => {
@@ -100,7 +118,6 @@ const internalAccessList = {
 					// Sanity check that something crazy hasn't happened
 					throw new error.InternalValidationError('Access List could not be updated, IDs do not match: ' + row.id + ' !== ' + data.id);
 				}
-
 			})
 			.then(() => {
 				// patch name if specified
@@ -109,7 +126,9 @@ const internalAccessList = {
 						.query()
 						.where({id: data.id})
 						.patch({
-							name: data.name
+							name:        data.name,
+							satisfy_any: data.satisfy_any,
+							pass_auth:   data.pass_auth,
 						});
 				}
 			})
@@ -153,6 +172,38 @@ const internalAccessList = {
 						});
 				}
 			})
+			.then(() => {
+				// Check for clients and add/update/remove them
+				if (typeof data.clients !== 'undefined' && data.clients) {
+					let promises = [];
+
+					data.clients.map(function (client) {
+						if (client.address) {
+							promises.push(accessListClientModel
+								.query()
+								.insert({
+									access_list_id: data.id,
+									address:        client.address,
+									directive:      client.directive
+								})
+							);
+						}
+					});
+
+					let query = accessListClientModel
+						.query()
+						.delete()
+						.where('access_list_id', data.id);
+
+					return query
+						.then(() => {
+							// Add new items
+							if (promises.length) {
+								return Promise.all(promises);
+							}
+						});
+				}
+			})
 			.then(() => {
 				// Add to audit log
 				return internalAuditLog.add(access, {
@@ -166,16 +217,16 @@ const internalAccessList = {
 				// re-fetch with expansions
 				return internalAccessList.get(access, {
 					id:     data.id,
-					expand: ['owner', 'items']
+					expand: ['owner', 'items', 'clients', 'proxy_hosts.[certificate,access_list.[clients,items]]']
 				}, true /* <- skip masking */);
 			})
 			.then((row) => {
 				return internalAccessList.build(row)
 					.then(() => {
 						if (row.proxy_host_count) {
-							return internalNginx.reload();
+							return internalNginx.bulkGenerateConfigs('proxy_host', row.proxy_hosts);
 						}
-					})
+					}).then(internalNginx.reload)
 					.then(() => {
 						return internalAccessList.maskItems(row);
 					});
@@ -204,35 +255,31 @@ const internalAccessList = {
 					.joinRaw('LEFT JOIN `proxy_host` ON `proxy_host`.`access_list_id` = `access_list`.`id` AND `proxy_host`.`is_deleted` = 0')
 					.where('access_list.is_deleted', 0)
 					.andWhere('access_list.id', data.id)
-					.allowEager('[owner,items,proxy_hosts]')
+					.allowGraph('[owner,items,clients,proxy_hosts.[certificate,access_list.[clients,items]]]')
-					.omit(['access_list.is_deleted'])
 					.first();
 
 				if (access_data.permission_visibility !== 'all') {
 					query.andWhere('access_list.owner_user_id', access.token.getUserId(1));
 				}
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
+					throw new error.ItemNotFoundError(data.id);
+				}
 				if (!skip_masking && typeof row.items !== 'undefined' && row.items) {
 					row = internalAccessList.maskItems(row);
 				}
-
+				// Custom omissions
-					return _.omit(row, omissions());
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-				} else {
+					row = _.omit(row, data.omit);
-					throw new error.ItemNotFoundError(data.id);
 				}
+				return row;
 			});
 	},
 
@@ -246,10 +293,10 @@ const internalAccessList = {
 	delete: (access, data) => {
 		return access.can('access_lists:delete', data.id)
 			.then(() => {
-				return internalAccessList.get(access, {id: data.id, expand: ['proxy_hosts', 'items']});
+				return internalAccessList.get(access, {id: data.id, expand: ['proxy_hosts', 'items', 'clients']});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				}
 
@@ -329,12 +376,11 @@ const internalAccessList = {
 					.joinRaw('LEFT JOIN `proxy_host` ON `proxy_host`.`access_list_id` = `access_list`.`id` AND `proxy_host`.`is_deleted` = 0')
 					.where('access_list.is_deleted', 0)
 					.groupBy('access_list.id')
-					.omit(['access_list.is_deleted'])
+					.allowGraph('[owner,items,clients]')
-					.allowEager('[owner,items]')
 					.orderBy('access_list.name', 'ASC');
 
 				if (access_data.permission_visibility !== 'all') {
-					query.andWhere('owner_user_id', access.token.getUserId(1));
+					query.andWhere('access_list.owner_user_id', access.token.getUserId(1));
 				}
 
 				// Query is used for searching
@@ -345,10 +391,10 @@ const internalAccessList = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			})
 			.then((rows) => {
 				if (rows) {
@@ -455,7 +501,7 @@ const internalAccessList = {
 								if (typeof item.password !== 'undefined' && item.password.length) {
 									logger.info('Adding: ' + item.username);
 
-									utils.exec('/usr/bin/htpasswd -b "' + htpasswd_file + '" "' + item.username + '" "' + item.password + '"')
+									utils.execFile('/usr/bin/htpasswd', ['-b', htpasswd_file, item.username, item.password])
 										.then((/*result*/) => {
 											next();
 										})

backend/internal/audit-log.js

@@ -19,7 +19,7 @@ const internalAuditLog = {
 					.orderBy('created_on', 'DESC')
 					.orderBy('id', 'DESC')
 					.limit(100)
-					.allowEager('[user]');
+					.allowGraph('[user]');
 
 				// Query is used for searching
 				if (typeof search_query === 'string') {
@@ -29,7 +29,7 @@ const internalAuditLog = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
 				return query;

backend/internal/certificate.js

@@ -1,18 +1,26 @@
-const fs               = require('fs');
 const _                = require('lodash');
-const logger           = require('../logger').ssl;
+const fs               = require('fs');
-const error            = require('../lib/error');
+const https            = require('https');
-const certificateModel = require('../models/certificate');
-const internalAuditLog = require('./audit-log');
 const tempWrite        = require('temp-write');
-const utils            = require('../lib/utils');
 const moment           = require('moment');
-const debug_mode       = process.env.NODE_ENV !== 'production' || !!process.env.DEBUG;
+const logger           = require('../logger').ssl;
-const le_staging       = process.env.NODE_ENV !== 'production';
+const config           = require('../lib/config');
+const error            = require('../lib/error');
+const utils            = require('../lib/utils');
+const certificateModel = require('../models/certificate');
+const tokenModel       = require('../models/token');
+const dnsPlugins       = require('../global/certbot-dns-plugins.json');
+const internalAuditLog = require('./audit-log');
 const internalNginx    = require('./nginx');
 const internalHost     = require('./host');
-const certbot_command  = '/usr/bin/certbot';
+const certbot          = require('../lib/certbot');
-const le_config        = '/etc/letsencrypt.ini';
+const archiver         = require('archiver');
+const path             = require('path');
+const { isArray }      = require('lodash');
+
+const letsencryptStaging = config.useLetsencryptStaging();
+const letsencryptConfig  = '/etc/letsencrypt.ini';
+const certbotCommand     = 'certbot';
 
 function omissions() {
 	return ['is_deleted'];
@@ -20,14 +28,15 @@ function omissions() {
 
 const internalCertificate = {
 
-	allowed_ssl_files:   ['certificate', 'certificate_key', 'intermediate_certificate'],
+	allowedSslFiles:         ['certificate', 'certificate_key', 'intermediate_certificate'],
-	interval_timeout:    1000 * 60 * 60, // 1 hour
+	intervalTimeout:         1000 * 60 * 60, // 1 hour
 	interval:                null,
-	interval_processing: false,
+	intervalProcessing:      false,
+	renewBeforeExpirationBy: [30, 'days'],
 
 	initTimer: () => {
 		logger.info('Let\'s Encrypt Renewal Timer initialized');
-		internalCertificate.interval = setInterval(internalCertificate.processExpiringHosts, internalCertificate.interval_timeout);
+		internalCertificate.interval = setInterval(internalCertificate.processExpiringHosts, internalCertificate.intervalTimeout);
 		// And do this now as well
 		internalCertificate.processExpiringHosts();
 	},
@@ -36,67 +45,58 @@ const internalCertificate = {
 	 * Triggered by a timer, this will check for expiring hosts and renew their ssl certs if required
 	 */
 	processExpiringHosts: () => {
-		if (!internalCertificate.interval_processing) {
+		if (!internalCertificate.intervalProcessing) {
-			internalCertificate.interval_processing = true;
+			internalCertificate.intervalProcessing = true;
-			logger.info('Renewing SSL certs close to expiry...');
+			logger.info('Renewing SSL certs expiring within ' + internalCertificate.renewBeforeExpirationBy[0] + ' ' + internalCertificate.renewBeforeExpirationBy[1] + ' ...');
 
-			let cmd = certbot_command + ' renew --non-interactive --quiet ' +
+			const expirationThreshold = moment().add(internalCertificate.renewBeforeExpirationBy[0], internalCertificate.renewBeforeExpirationBy[1]).format('YYYY-MM-DD HH:mm:ss');
-				'--config "' + le_config + '" ' +
-				'--preferred-challenges "dns,http" ' +
-				'--disable-hook-validation ' +
-				(le_staging ? '--staging' : '');
 
-			return utils.exec(cmd)
+			// Fetch all the letsencrypt certs from the db that will expire within the configured threshold
-				.then((result) => {
+			certificateModel
-					if (result) {
-						logger.info('Renew Result: ' + result);
-					}
-
-					return internalNginx.reload()
-						.then(() => {
-							logger.info('Renew Complete');
-							return result;
-						});
-				})
-				.then(() => {
-					// Now go and fetch all the letsencrypt certs from the db and query the files and update expiry times
-					return certificateModel
 				.query()
 				.where('is_deleted', 0)
 				.andWhere('provider', 'letsencrypt')
+				.andWhere('expires_on', '<', expirationThreshold)
 				.then((certificates) => {
-							if (certificates && certificates.length) {
+					if (!certificates || !certificates.length) {
-								let promises = [];
+						return null;
+					}
 
-								certificates.map(function (certificate) {
+					/**
-									promises.push(
+					 * Renews must be run sequentially or we'll get an error 'Another
-										internalCertificate.getCertificateInfoFromFile('/etc/letsencrypt/live/npm-' + certificate.id + '/fullchain.pem')
+					 * instance of Certbot is already running.'
-											.then((cert_info) => {
+					 */
-												return certificateModel
+					let sequence = Promise.resolve();
-													.query()
+
-													.where('id', certificate.id)
+					certificates.forEach(function (certificate) {
-													.andWhere('provider', 'letsencrypt')
+						sequence = sequence.then(() =>
-													.patch({
+							internalCertificate
-														expires_on: certificateModel.raw('FROM_UNIXTIME(' + cert_info.dates.to + ')')
+								.renew(
-													});
+									{
-											})
+										can: () =>
+											Promise.resolve({
+												permission_visibility: 'all',
+											}),
+										token: new tokenModel(),
+									},
+									{ id: certificate.id },
+								)
 								.catch((err) => {
 									// Don't want to stop the train here, just log the error
 									logger.error(err.message);
-											})
+								}),
 						);
 					});
 
-								return Promise.all(promises);
+					return sequence;
-							}
-						});
 				})
 				.then(() => {
-					internalCertificate.interval_processing = false;
+					logger.info('Completed SSL cert renew process');
+					internalCertificate.intervalProcessing = false;
 				})
 				.catch((err) => {
 					logger.error(err);
-					internalCertificate.interval_processing = false;
+					internalCertificate.intervalProcessing = false;
 				});
 		}
 	},
@@ -112,13 +112,13 @@ const internalCertificate = {
 				data.owner_user_id = access.token.getUserId(1);
 
 				if (data.provider === 'letsencrypt') {
-					data.nice_name = data.domain_names.sort().join(', ');
+					data.nice_name = data.domain_names.join(', ');
 				}
 
 				return certificateModel
 					.query()
-					.omit(omissions())
+					.insertAndFetch(data)
-					.insertAndFetch(data);
+					.then(utils.omitRow(omissions()));
 			})
 			.then((certificate) => {
 				if (certificate.provider === 'letsencrypt') {
@@ -141,9 +141,33 @@ const internalCertificate = {
 								});
 						})
 						.then((in_use_result) => {
+							// With DNS challenge no config is needed, so skip 3 and 5.
+							if (certificate.meta.dns_challenge) {
+								return internalNginx.reload().then(() => {
+									// 4. Request cert
+									return internalCertificate.requestLetsEncryptSslWithDnsChallenge(certificate);
+								})
+									.then(internalNginx.reload)
+									.then(() => {
+										// 6. Re-instate previously disabled hosts
+										return internalCertificate.enableInUseHosts(in_use_result);
+									})
+									.then(() => {
+										return certificate;
+									})
+									.catch((err) => {
+										// In the event of failure, revert things and throw err back
+										return internalCertificate.enableInUseHosts(in_use_result)
+											.then(internalNginx.reload)
+											.then(() => {
+												throw err;
+											});
+									});
+							} else {
 								// 3. Generate the LE config
 								return internalNginx.generateLetsEncryptRequestConfig(certificate)
 									.then(internalNginx.reload)
+									.then(async() => await new Promise((r) => setTimeout(r, 5000)))
 									.then(() => {
 										// 4. Request cert
 										return internalCertificate.requestLetsEncryptSsl(certificate);
@@ -171,6 +195,7 @@ const internalCertificate = {
 												throw err;
 											});
 									});
+							}
 						})
 						.then(() => {
 							// At this point, the letsencrypt cert should exist on disk.
@@ -180,7 +205,7 @@ const internalCertificate = {
 									return certificateModel
 										.query()
 										.patchAndFetchById(certificate.id, {
-											expires_on: certificateModel.raw('FROM_UNIXTIME(' + cert_info.dates.to + ')')
+											expires_on: moment(cert_info.dates.to, 'X').format('YYYY-MM-DD HH:mm:ss')
 										})
 										.then((saved_row) => {
 											// Add cert data for audit log
@@ -191,6 +216,13 @@ const internalCertificate = {
 											return saved_row;
 										});
 								});
+						}).catch(async (error) => {
+							// Delete the certificate from the database if it was not created successfully
+							await certificateModel
+								.query()
+								.deleteById(certificate.id);
+
+							throw error;
 						});
 				} else {
 					return certificate;
@@ -233,8 +265,8 @@ const internalCertificate = {
 
 				return certificateModel
 					.query()
-					.omit(omissions())
 					.patchAndFetchById(row.id, data)
+					.then(utils.omitRow(omissions()))
 					.then((saved_row) => {
 						saved_row.meta = internalCertificate.cleanMeta(saved_row.meta);
 						data.meta      = internalCertificate.cleanMeta(data.meta);
@@ -252,7 +284,7 @@ const internalCertificate = {
 							meta:        _.omit(data, ['expires_on']) // this prevents json circular reference because expires_on might be raw
 						})
 							.then(() => {
-								return _.omit(saved_row, omissions());
+								return saved_row;
 							});
 					});
 			});
@@ -277,30 +309,93 @@ const internalCertificate = {
 					.query()
 					.where('is_deleted', 0)
 					.andWhere('id', data.id)
-					.allowEager('[owner]')
+					.allowGraph('[owner]')
 					.first();
 
 				if (access_data.permission_visibility !== 'all') {
 					query.andWhere('owner_user_id', access.token.getUserId(1));
 				}
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
-					return _.omit(row, omissions());
-				} else {
 					throw new error.ItemNotFoundError(data.id);
 				}
+				// Custom omissions
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
+					row = _.omit(row, data.omit);
+				}
+				return row;
+			});
+	},
+
+	/**
+	 * @param   {Access}  access
+	 * @param   {Object}  data
+	 * @param   {Number}  data.id
+	 * @returns {Promise}
+	 */
+	download: (access, data) => {
+		return new Promise((resolve, reject) => {
+			access.can('certificates:get', data)
+				.then(() => {
+					return internalCertificate.get(access, data);
+				})
+				.then((certificate) => {
+					if (certificate.provider === 'letsencrypt') {
+						const zipDirectory = '/etc/letsencrypt/live/npm-' + data.id;
+
+						if (!fs.existsSync(zipDirectory)) {
+							throw new error.ItemNotFoundError('Certificate ' + certificate.nice_name + ' does not exists');
+						}
+
+						let certFiles      = fs.readdirSync(zipDirectory)
+							.filter((fn) => fn.endsWith('.pem'))
+							.map((fn) => fs.realpathSync(path.join(zipDirectory, fn)));
+						const downloadName = 'npm-' + data.id + '-' + `${Date.now()}.zip`;
+						const opName       = '/tmp/' + downloadName;
+						internalCertificate.zipFiles(certFiles, opName)
+							.then(() => {
+								logger.debug('zip completed : ', opName);
+								const resp = {
+									fileName: opName
+								};
+								resolve(resp);
+							}).catch((err) => reject(err));
+					} else {
+						throw new error.ValidationError('Only Let\'sEncrypt certificates can be downloaded');
+					}
+				}).catch((err) => reject(err));
+		});
+	},
+
+	/**
+	* @param   {String}  source
+	* @param   {String}  out
+	* @returns {Promise}
+	*/
+	zipFiles(source, out) {
+		const archive = archiver('zip', { zlib: { level: 9 } });
+		const stream  = fs.createWriteStream(out);
+
+		return new Promise((resolve, reject) => {
+			source
+				.map((fl) => {
+					let fileName = path.basename(fl);
+					logger.debug(fl, 'added to certificate zip');
+					archive.file(fl, { name: fileName });
+				});
+			archive
+				.on('error', (err) => reject(err))
+				.pipe(stream);
+
+			stream.on('close', () => resolve());
+			archive.finalize();
 		});
 	},
 
@@ -317,7 +412,7 @@ const internalCertificate = {
 				return internalCertificate.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				}
 
@@ -365,8 +460,7 @@ const internalCertificate = {
 					.query()
 					.where('is_deleted', 0)
 					.groupBy('id')
-					.omit(['is_deleted'])
+					.allowGraph('[owner]')
-					.allowEager('[owner]')
 					.orderBy('nice_name', 'ASC');
 
 				if (access_data.permission_visibility !== 'all') {
@@ -376,15 +470,15 @@ const internalCertificate = {
 				// Query is used for searching
 				if (typeof search_query === 'string') {
 					query.where(function () {
-						this.where('name', 'like', '%' + search_query + '%');
+						this.where('nice_name', 'like', '%' + search_query + '%');
 					});
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			});
 	},
 
@@ -416,11 +510,9 @@ const internalCertificate = {
 	 * @returns {Promise}
 	 */
 	writeCustomCert: (certificate) => {
-		if (debug_mode) {
 		logger.info('Writing Custom Certificate:', certificate);
-		}
 
-		let dir = '/data/custom_ssl/npm-' + certificate.id;
+		const dir = '/data/custom_ssl/npm-' + certificate.id;
 
 		return new Promise((resolve, reject) => {
 			if (certificate.provider === 'letsencrypt') {
@@ -428,9 +520,9 @@ const internalCertificate = {
 				return;
 			}
 
-			let cert_data = certificate.meta.certificate;
+			let certData = certificate.meta.certificate;
 			if (typeof certificate.meta.intermediate_certificate !== 'undefined') {
-				cert_data = cert_data + '\n' + certificate.meta.intermediate_certificate;
+				certData = certData + '\n' + certificate.meta.intermediate_certificate;
 			}
 
 			try {
@@ -442,7 +534,7 @@ const internalCertificate = {
 				return;
 			}
 
-			fs.writeFile(dir + '/fullchain.pem', cert_data, function (err) {
+			fs.writeFile(dir + '/fullchain.pem', certData, function (err) {
 				if (err) {
 					reject(err);
 				} else {
@@ -492,7 +584,7 @@ const internalCertificate = {
 			// Put file contents into an object
 			let files = {};
 			_.map(data.files, (file, name) => {
-				if (internalCertificate.allowed_ssl_files.indexOf(name) !== -1) {
+				if (internalCertificate.allowedSslFiles.indexOf(name) !== -1) {
 					files[name] = file.data.toString();
 				}
 			});
@@ -550,7 +642,7 @@ const internalCertificate = {
 						}
 
 						_.map(data.files, (file, name) => {
-							if (internalCertificate.allowed_ssl_files.indexOf(name) !== -1) {
+							if (internalCertificate.allowedSslFiles.indexOf(name) !== -1) {
 								row.meta[name] = file.data.toString();
 							}
 						});
@@ -558,18 +650,17 @@ const internalCertificate = {
 						// TODO: This uses a mysql only raw function that won't translate to postgres
 						return internalCertificate.update(access, {
 							id:           data.id,
-							expires_on:   certificateModel.raw('FROM_UNIXTIME(' + validations.certificate.dates.to + ')'),
+							expires_on:   moment(validations.certificate.dates.to, 'X').format('YYYY-MM-DD HH:mm:ss'),
 							domain_names: [validations.certificate.cn],
 							meta:         _.clone(row.meta) // Prevent the update method from changing this value that we'll use later
 						})
 							.then((certificate) => {
-								console.log('ROWMETA:', row.meta);
 								certificate.meta = row.meta;
 								return internalCertificate.writeCustomCert(certificate);
 							});
 					})
 					.then(() => {
-						return _.pick(row.meta, internalCertificate.allowed_ssl_files);
+						return _.pick(row.meta, internalCertificate.allowedSslFiles);
 					});
 			});
 	},
@@ -583,17 +674,25 @@ const internalCertificate = {
 	checkPrivateKey: (private_key) => {
 		return tempWrite(private_key, '/tmp')
 			.then((filepath) => {
-				return utils.exec('openssl rsa -in ' + filepath + ' -check -noout')
+				return new Promise((resolve, reject) => {
+					const failTimeout = setTimeout(() => {
+						reject(new error.ValidationError('Result Validation Error: Validation timed out. This could be due to the key being passphrase-protected.'));
+					}, 10000);
+					utils
+						.exec('openssl pkey -in ' + filepath + ' -check -noout 2>&1 ')
 						.then((result) => {
-						if (!result.toLowerCase().includes('key ok')) {
+							clearTimeout(failTimeout);
-							throw new error.ValidationError(result);
+							if (!result.toLowerCase().includes('key is valid')) {
+								reject(new error.ValidationError('Result Validation Error: ' + result));
 							}
-
 							fs.unlinkSync(filepath);
-						return true;
+							resolve(true);
-					}).catch((err) => {
+						})
+						.catch((err) => {
+							clearTimeout(failTimeout);
 							fs.unlinkSync(filepath);
-						throw new error.ValidationError('Certificate Key is not valid (' + err.message + ')', err);
+							reject(new error.ValidationError('Certificate Key is not valid (' + err.message + ')', err));
+						});
 				});
 			});
 	},
@@ -609,9 +708,9 @@ const internalCertificate = {
 		return tempWrite(certificate, '/tmp')
 			.then((filepath) => {
 				return internalCertificate.getCertificateInfoFromFile(filepath, throw_expired)
-					.then((cert_data) => {
+					.then((certData) => {
 						fs.unlinkSync(filepath);
-						return cert_data;
+						return certData;
 					}).catch((err) => {
 						fs.unlinkSync(filepath);
 						throw err;
@@ -627,33 +726,33 @@ const internalCertificate = {
 	 * @param {Boolean} [throw_expired]  Throw when the certificate is out of date
 	 */
 	getCertificateInfoFromFile: (certificate_file, throw_expired) => {
-		let cert_data = {};
+		let certData = {};
 
 		return utils.exec('openssl x509 -in ' + certificate_file + ' -subject -noout')
 			.then((result) => {
 				// subject=CN = something.example.com
-				let regex = /(?:subject=)?[^=]+=\s+(\S+)/gim;
+				const regex = /(?:subject=)?[^=]+=\s+(\S+)/gim;
-				let match = regex.exec(result);
+				const match = regex.exec(result);
 
 				if (typeof match[1] === 'undefined') {
 					throw new error.ValidationError('Could not determine subject from certificate: ' + result);
 				}
 
-				cert_data['cn'] = match[1];
+				certData['cn'] = match[1];
 			})
 			.then(() => {
 				return utils.exec('openssl x509 -in ' + certificate_file + ' -issuer -noout');
 			})
 			.then((result) => {
 				// issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
-				let regex = /^(?:issuer=)?(.*)$/gim;
+				const regex = /^(?:issuer=)?(.*)$/gim;
-				let match = regex.exec(result);
+				const match = regex.exec(result);
 
 				if (typeof match[1] === 'undefined') {
 					throw new error.ValidationError('Could not determine issuer from certificate: ' + result);
 				}
 
-				cert_data['issuer'] = match[1];
+				certData['issuer'] = match[1];
 			})
 			.then(() => {
 				return utils.exec('openssl x509 -in ' + certificate_file + ' -dates -noout');
@@ -661,39 +760,39 @@ const internalCertificate = {
 			.then((result) => {
 				// notBefore=Jul 14 04:04:29 2018 GMT
 				// notAfter=Oct 12 04:04:29 2018 GMT
-				let valid_from = null;
+				let validFrom = null;
-				let valid_to   = null;
+				let validTo   = null;
 
-				let lines = result.split('\n');
+				const lines = result.split('\n');
 				lines.map(function (str) {
-					let regex = /^(\S+)=(.*)$/gim;
+					const regex = /^(\S+)=(.*)$/gim;
-					let match = regex.exec(str.trim());
+					const match = regex.exec(str.trim());
 
 					if (match && typeof match[2] !== 'undefined') {
-						let date = parseInt(moment(match[2], 'MMM DD HH:mm:ss YYYY z').format('X'), 10);
+						const date = parseInt(moment(match[2], 'MMM DD HH:mm:ss YYYY z').format('X'), 10);
 
 						if (match[1].toLowerCase() === 'notbefore') {
-							valid_from = date;
+							validFrom = date;
 						} else if (match[1].toLowerCase() === 'notafter') {
-							valid_to = date;
+							validTo = date;
 						}
 					}
 				});
 
-				if (!valid_from || !valid_to) {
+				if (!validFrom || !validTo) {
 					throw new error.ValidationError('Could not determine dates from certificate: ' + result);
 				}
 
-				if (throw_expired && valid_to < parseInt(moment().format('X'), 10)) {
+				if (throw_expired && validTo < parseInt(moment().format('X'), 10)) {
 					throw new error.ValidationError('Certificate has expired');
 				}
 
-				cert_data['dates'] = {
+				certData['dates'] = {
-					from: valid_from,
+					from: validFrom,
-					to:   valid_to
+					to:   validTo
 				};
 
-				return cert_data;
+				return certData;
 			}).catch((err) => {
 				throw new error.ValidationError('Certificate is not valid (' + err.message + ')', err);
 			});
@@ -707,7 +806,7 @@ const internalCertificate = {
 	 * @returns {Object}
 	 */
 	cleanMeta: function (meta, remove) {
-		internalCertificate.allowed_ssl_files.map((key) => {
+		internalCertificate.allowedSslFiles.map((key) => {
 			if (typeof meta[key] !== 'undefined' && meta[key]) {
 				if (remove) {
 					delete meta[key];
@@ -721,25 +820,26 @@ const internalCertificate = {
 	},
 
 	/**
+	 * Request a certificate using the http challenge
 	 * @param   {Object}  certificate   the certificate row
 	 * @returns {Promise}
 	 */
 	requestLetsEncryptSsl: (certificate) => {
 		logger.info('Requesting Let\'sEncrypt certificates for Cert #' + certificate.id + ': ' + certificate.domain_names.join(', '));
 
-		let cmd = certbot_command + ' certonly --non-interactive ' +
+		const cmd = certbotCommand + ' certonly ' +
-			'--config "' + le_config + '" ' +
+			'--config "' + letsencryptConfig + '" ' +
+			'--work-dir "/tmp/letsencrypt-lib" ' +
+			'--logs-dir "/tmp/letsencrypt-log" ' +
 			'--cert-name "npm-' + certificate.id + '" ' +
 			'--agree-tos ' +
+			'--authenticator webroot ' +
 			'--email "' + certificate.meta.letsencrypt_email + '" ' +
 			'--preferred-challenges "dns,http" ' +
-			'--webroot ' +
 			'--domains "' + certificate.domain_names.join(',') + '" ' +
-			(le_staging ? '--staging' : '');
+			(letsencryptStaging ? '--staging' : '');
 
-		if (debug_mode) {
 		logger.info('Command:', cmd);
-		}
 
 		return utils.exec(cmd)
 			.then((result) => {
@@ -748,6 +848,69 @@ const internalCertificate = {
 			});
 	},
 
+	/**
+	 * @param   {Object}         certificate          the certificate row
+	 * @param   {String}         dns_provider         the dns provider name (key used in `certbot-dns-plugins.json`)
+	 * @param   {String | null}  credentials          the content of this providers credentials file
+	 * @param   {String}         propagation_seconds
+	 * @returns {Promise}
+	 */
+	requestLetsEncryptSslWithDnsChallenge: async (certificate) => {
+		await certbot.installPlugin(certificate.meta.dns_provider);
+		const dnsPlugin = dnsPlugins[certificate.meta.dns_provider];
+		logger.info(`Requesting Let'sEncrypt certificates via ${dnsPlugin.name} for Cert #${certificate.id}: ${certificate.domain_names.join(', ')}`);
+
+		const credentialsLocation = '/etc/letsencrypt/credentials/credentials-' + certificate.id;
+		fs.mkdirSync('/etc/letsencrypt/credentials', { recursive: true });
+		fs.writeFileSync(credentialsLocation, certificate.meta.dns_provider_credentials, {mode: 0o600});
+
+		// Whether the plugin has a --<name>-credentials argument
+		const hasConfigArg = certificate.meta.dns_provider !== 'route53';
+
+		let mainCmd = certbotCommand + ' certonly ' +
+			'--config "' + letsencryptConfig + '" ' +
+			'--work-dir "/tmp/letsencrypt-lib" ' +
+			'--logs-dir "/tmp/letsencrypt-log" ' +
+			'--cert-name "npm-' + certificate.id + '" ' +
+			'--agree-tos ' +
+			'--email "' + certificate.meta.letsencrypt_email + '" ' +
+			'--domains "' + certificate.domain_names.join(',') + '" ' +
+			'--authenticator ' + dnsPlugin.full_plugin_name + ' ' +
+			(
+				hasConfigArg
+					? '--' + dnsPlugin.full_plugin_name + '-credentials "' + credentialsLocation + '"'
+					: ''
+			) +
+			(
+				certificate.meta.propagation_seconds !== undefined
+					? ' --' + dnsPlugin.full_plugin_name + '-propagation-seconds ' + certificate.meta.propagation_seconds
+					: ''
+			) +
+			(letsencryptStaging ? ' --staging' : '');
+
+		// Prepend the path to the credentials file as an environment variable
+		if (certificate.meta.dns_provider === 'route53') {
+			mainCmd = 'AWS_CONFIG_FILE=\'' + credentialsLocation + '\' ' + mainCmd;
+		}
+
+		if (certificate.meta.dns_provider === 'duckdns') {
+			mainCmd = mainCmd + ' --dns-duckdns-no-txt-restore';
+		}
+
+		logger.info('Command:', mainCmd);
+
+		try {
+			const result = await utils.exec(mainCmd);
+			logger.info(result);
+			return result;
+		} catch (err) {
+			// Don't fail if file does not exist, so no need for action in the callback
+			fs.unlink(credentialsLocation, () => {});
+			throw err;
+		}
+	},
+
+
 	/**
 	 * @param   {Access}  access
 	 * @param   {Object}  data
@@ -761,7 +924,9 @@ const internalCertificate = {
 			})
 			.then((certificate) => {
 				if (certificate.provider === 'letsencrypt') {
-					return internalCertificate.renewLetsEncryptSsl(certificate)
+					const renewMethod = certificate.meta.dns_challenge ? internalCertificate.renewLetsEncryptSslWithDnsChallenge : internalCertificate.renewLetsEncryptSsl;
+
+					return renewMethod(certificate)
 						.then(() => {
 							return internalCertificate.getCertificateInfoFromFile('/etc/letsencrypt/live/npm-' + certificate.id + '/fullchain.pem');
 						})
@@ -769,7 +934,7 @@ const internalCertificate = {
 							return certificateModel
 								.query()
 								.patchAndFetchById(certificate.id, {
-									expires_on: certificateModel.raw('FROM_UNIXTIME(' + cert_info.dates.to + ')')
+									expires_on: moment(cert_info.dates.to, 'X').format('YYYY-MM-DD HH:mm:ss')
 								});
 						})
 						.then((updated_certificate) => {
@@ -797,16 +962,17 @@ const internalCertificate = {
 	renewLetsEncryptSsl: (certificate) => {
 		logger.info('Renewing Let\'sEncrypt certificates for Cert #' + certificate.id + ': ' + certificate.domain_names.join(', '));
 
-		let cmd = certbot_command + ' renew --non-interactive ' +
+		const cmd = certbotCommand + ' renew --force-renewal ' +
-			'--config "' + le_config + '" ' +
+			'--config "' + letsencryptConfig + '" ' +
+			'--work-dir "/tmp/letsencrypt-lib" ' +
+			'--logs-dir "/tmp/letsencrypt-log" ' +
 			'--cert-name "npm-' + certificate.id + '" ' +
 			'--preferred-challenges "dns,http" ' +
+			'--no-random-sleep-on-renew ' +
 			'--disable-hook-validation ' +
-			(le_staging ? '--staging' : '');
+			(letsencryptStaging ? '--staging' : '');
 
-		if (debug_mode) {
 		logger.info('Command:', cmd);
-		}
 
 		return utils.exec(cmd)
 			.then((result) => {
@@ -815,6 +981,43 @@ const internalCertificate = {
 			});
 	},
 
+	/**
+	 * @param   {Object}  certificate   the certificate row
+	 * @returns {Promise}
+	 */
+	renewLetsEncryptSslWithDnsChallenge: (certificate) => {
+		const dnsPlugin = dnsPlugins[certificate.meta.dns_provider];
+
+		if (!dnsPlugin) {
+			throw Error(`Unknown DNS provider '${certificate.meta.dns_provider}'`);
+		}
+
+		logger.info(`Renewing Let'sEncrypt certificates via ${dnsPlugin.name} for Cert #${certificate.id}: ${certificate.domain_names.join(', ')}`);
+
+		let mainCmd = certbotCommand + ' renew --force-renewal ' +
+			'--config "' + letsencryptConfig + '" ' +
+			'--work-dir "/tmp/letsencrypt-lib" ' +
+			'--logs-dir "/tmp/letsencrypt-log" ' +
+			'--cert-name "npm-' + certificate.id + '" ' +
+			'--disable-hook-validation ' +
+			'--no-random-sleep-on-renew ' +
+			(letsencryptStaging ? ' --staging' : '');
+
+		// Prepend the path to the credentials file as an environment variable
+		if (certificate.meta.dns_provider === 'route53') {
+			const credentialsLocation = '/etc/letsencrypt/credentials/credentials-' + certificate.id;
+			mainCmd                   = 'AWS_CONFIG_FILE=\'' + credentialsLocation + '\' ' + mainCmd;
+		}
+
+		logger.info('Command:', mainCmd);
+
+		return utils.exec(mainCmd)
+			.then(async (result) => {
+				logger.info(result);
+				return result;
+			});
+	},
+
 	/**
 	 * @param   {Object}  certificate    the certificate row
 	 * @param   {Boolean} [throw_errors]
@@ -823,28 +1026,27 @@ const internalCertificate = {
 	revokeLetsEncryptSsl: (certificate, throw_errors) => {
 		logger.info('Revoking Let\'sEncrypt certificates for Cert #' + certificate.id + ': ' + certificate.domain_names.join(', '));
 
-		let cmd = certbot_command + ' revoke --non-interactive ' +
+		const mainCmd = certbotCommand + ' revoke ' +
-			'--config "' + le_config + '" ' +
+			'--config "' + letsencryptConfig + '" ' +
+			'--work-dir "/tmp/letsencrypt-lib" ' +
+			'--logs-dir "/tmp/letsencrypt-log" ' +
 			'--cert-path "/etc/letsencrypt/live/npm-' + certificate.id + '/fullchain.pem" ' +
 			'--delete-after-revoke ' +
-			(le_staging ? '--staging' : '');
+			(letsencryptStaging ? '--staging' : '');
 
-		if (debug_mode) {
+		// Don't fail command if file does not exist
-			logger.info('Command:', cmd);
+		const delete_credentialsCmd = `rm -f '/etc/letsencrypt/credentials/credentials-${certificate.id}' || true`;
-		}
 
-		return utils.exec(cmd)
+		logger.info('Command:', mainCmd + '; ' + delete_credentialsCmd);
-			.then((result) => {
+
-				if (debug_mode) {
+		return utils.exec(mainCmd)
-					logger.info('Command:', cmd);
+			.then(async (result) => {
-				}
+				await utils.exec(delete_credentialsCmd);
 				logger.info(result);
 				return result;
 			})
 			.catch((err) => {
-				if (debug_mode) {
 				logger.error(err.message);
-				}
 
 				if (throw_errors) {
 					throw err;
@@ -857,9 +1059,9 @@ const internalCertificate = {
 	 * @returns {Boolean}
 	 */
 	hasLetsEncryptSslCerts: (certificate) => {
-		let le_path = '/etc/letsencrypt/live/npm-' + certificate.id;
+		const letsencryptPath = '/etc/letsencrypt/live/npm-' + certificate.id;
 
-		return fs.existsSync(le_path + '/fullchain.pem') && fs.existsSync(le_path + '/privkey.pem');
+		return fs.existsSync(letsencryptPath + '/fullchain.pem') && fs.existsSync(letsencryptPath + '/privkey.pem');
 	},
 
 	/**
@@ -920,6 +1122,108 @@ const internalCertificate = {
 		} else {
 			return Promise.resolve();
 		}
+	},
+
+	testHttpsChallenge: async (access, domains) => {
+		await access.can('certificates:list');
+
+		if (!isArray(domains)) {
+			throw new error.InternalValidationError('Domains must be an array of strings');
+		}
+		if (domains.length === 0) {
+			throw new error.InternalValidationError('No domains provided');
+		}
+
+		// Create a test challenge file
+		const testChallengeDir  = '/data/letsencrypt-acme-challenge/.well-known/acme-challenge';
+		const testChallengeFile = testChallengeDir + '/test-challenge';
+		fs.mkdirSync(testChallengeDir, {recursive: true});
+		fs.writeFileSync(testChallengeFile, 'Success', {encoding: 'utf8'});
+
+		async function performTestForDomain (domain) {
+			logger.info('Testing http challenge for ' + domain);
+			const url      = `http://${domain}/.well-known/acme-challenge/test-challenge`;
+			const formBody = `method=G&url=${encodeURI(url)}&bodytype=T&requestbody=&headername=User-Agent&headervalue=None&locationid=1&ch=false&cc=false`;
+			const options  = {
+				method:  'POST',
+				headers: {
+					'User-Agent':     'Mozilla/5.0',
+					'Content-Type':   'application/x-www-form-urlencoded',
+					'Content-Length': Buffer.byteLength(formBody)
+				}
+			};
+
+			const result = await new Promise((resolve) => {
+
+				const req = https.request('https://www.site24x7.com/tools/restapi-tester', options, function (res) {
+					let responseBody = '';
+
+					res.on('data', (chunk) => responseBody = responseBody + chunk);
+					res.on('end', function () {
+						try {
+							const parsedBody = JSON.parse(responseBody + '');
+							if (res.statusCode !== 200) {
+								logger.warn(`Failed to test HTTP challenge for domain ${domain} because HTTP status code ${res.statusCode} was returned: ${parsedBody.message}`);
+								resolve(undefined);
+							} else {
+								resolve(parsedBody);
+							}
+						} catch (err) {
+							if (res.statusCode !== 200) {
+								logger.warn(`Failed to test HTTP challenge for domain ${domain} because HTTP status code ${res.statusCode} was returned`);
+							} else {
+								logger.warn(`Failed to test HTTP challenge for domain ${domain} because response failed to be parsed: ${err.message}`);
+							}
+							resolve(undefined);
+						}
+					});
+				});
+
+				// Make sure to write the request body.
+				req.write(formBody);
+				req.end();
+				req.on('error', function (e) { logger.warn(`Failed to test HTTP challenge for domain ${domain}`, e);
+					resolve(undefined); });
+			});
+
+			if (!result) {
+				// Some error occurred while trying to get the data
+				return 'failed';
+			} else if (result.error) {
+				logger.info(`HTTP challenge test failed for domain ${domain} because error was returned: ${result.error.msg}`);
+				return `other:${result.error.msg}`;
+			} else if (`${result.responsecode}` === '200' && result.htmlresponse === 'Success') {
+				// Server exists and has responded with the correct data
+				return 'ok';
+			} else if (`${result.responsecode}` === '200') {
+				// Server exists but has responded with wrong data
+				logger.info(`HTTP challenge test failed for domain ${domain} because of invalid returned data:`, result.htmlresponse);
+				return 'wrong-data';
+			} else if (`${result.responsecode}` === '404') {
+				// Server exists but responded with a 404
+				logger.info(`HTTP challenge test failed for domain ${domain} because code 404 was returned`);
+				return '404';
+			} else if (`${result.responsecode}` === '0' || (typeof result.reason === 'string' && result.reason.toLowerCase() === 'host unavailable')) {
+				// Server does not exist at domain
+				logger.info(`HTTP challenge test failed for domain ${domain} the host was not found`);
+				return 'no-host';
+			} else {
+				// Other errors
+				logger.info(`HTTP challenge test failed for domain ${domain} because code ${result.responsecode} was returned`);
+				return `other:${result.responsecode}`;
+			}
+		}
+
+		const results = {};
+
+		for (const domain of domains){
+			results[domain] = await performTestForDomain(domain);
+		}
+
+		// Remove the test challenge file
+		fs.unlinkSync(testChallengeFile);
+
+		return results;
 	}
 };
 

backend/internal/dead-host.js

@@ -1,5 +1,6 @@
 const _                   = require('lodash');
 const error               = require('../lib/error');
+const utils               = require('../lib/utils');
 const deadHostModel       = require('../models/dead_host');
 const internalHost        = require('./host');
 const internalNginx       = require('./nginx');
@@ -47,10 +48,16 @@ const internalDeadHost = {
 				data.owner_user_id = access.token.getUserId(1);
 				data               = internalHost.cleanSslHstsData(data);
 
+				// Fix for db field not having a default value
+				// for this optional field.
+				if (typeof data.advanced_config === 'undefined') {
+					data.advanced_config = '';
+				}
+
 				return deadHostModel
 					.query()
-					.omit(omissions())
+					.insertAndFetch(data)
-					.insertAndFetch(data);
+					.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
 				if (create_certificate) {
@@ -218,31 +225,28 @@ const internalDeadHost = {
 					.query()
 					.where('is_deleted', 0)
 					.andWhere('id', data.id)
-					.allowEager('[owner,certificate]')
+					.allowGraph('[owner,certificate]')
 					.first();
 
 				if (access_data.permission_visibility !== 'all') {
 					query.andWhere('owner_user_id', access.token.getUserId(1));
 				}
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
-					row = internalHost.cleanRowCertificateMeta(row);
-					return _.omit(row, omissions());
-				} else {
 					throw new error.ItemNotFoundError(data.id);
 				}
+				// Custom omissions
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
+					row = _.omit(row, data.omit);
+				}
+				return row;
 			});
 	},
 
@@ -259,7 +263,7 @@ const internalDeadHost = {
 				return internalDeadHost.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				}
 
@@ -307,7 +311,7 @@ const internalDeadHost = {
 				});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (row.enabled) {
 					throw new error.ValidationError('Host is already enabled');
@@ -353,7 +357,7 @@ const internalDeadHost = {
 				return internalDeadHost.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (!row.enabled) {
 					throw new error.ValidationError('Host is already disabled');
@@ -404,8 +408,7 @@ const internalDeadHost = {
 					.query()
 					.where('is_deleted', 0)
 					.groupBy('id')
-					.omit(['is_deleted'])
+					.allowGraph('[owner,certificate]')
-					.allowEager('[owner,certificate]')
 					.orderBy('domain_names', 'ASC');
 
 				if (access_data.permission_visibility !== 'all') {
@@ -420,10 +423,10 @@ const internalDeadHost = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			})
 			.then((rows) => {
 				if (typeof expand !== 'undefined' && expand !== null && expand.indexOf('certificate') !== -1) {

backend/internal/host.js

@@ -106,7 +106,7 @@ const internalHost = {
 					response_object.total_count      += response_object.redirection_hosts.length;
 				}
 
-				if (promises_results[1]) {
+				if (promises_results[2]) {
 					// Dead Hosts
 					response_object.dead_hosts   = internalHost._getHostsWithDomains(promises_results[2], domain_names);
 					response_object.total_count += response_object.dead_hosts.length;
@@ -158,7 +158,7 @@ const internalHost = {
 					}
 				}
 
-				if (promises_results[1]) {
+				if (promises_results[2]) {
 					// Dead Hosts
 					if (internalHost._checkHostnameRecordsTaken(hostname, promises_results[2], ignore_type === 'dead' && ignore_id ? ignore_id : 0)) {
 						is_taken = true;

backend/internal/ip_ranges.js

@@ -2,13 +2,16 @@ const https         = require('https');
 const fs            = require('fs');
 const logger        = require('../logger').ip_ranges;
 const error         = require('../lib/error');
+const utils         = require('../lib/utils');
 const internalNginx = require('./nginx');
-const { Liquid }    = require('liquidjs');
 
 const CLOUDFRONT_URL   = 'https://ip-ranges.amazonaws.com/ip-ranges.json';
 const CLOUDFARE_V4_URL = 'https://www.cloudflare.com/ips-v4';
 const CLOUDFARE_V6_URL = 'https://www.cloudflare.com/ips-v6';
 
+const regIpV4 = /^(\d+\.?){4}\/\d+/;
+const regIpV6 = /^(([\da-fA-F]+)?:)+\/\d+/;
+
 const internalIpRanges = {
 
 	interval_timeout:    1000 * 60 * 60 * 6, // 6 hours
@@ -74,14 +77,14 @@ const internalIpRanges = {
 					return internalIpRanges.fetchUrl(CLOUDFARE_V4_URL);
 				})
 				.then((cloudfare_data) => {
-					let items = cloudfare_data.split('\n');
+					let items = cloudfare_data.split('\n').filter((line) => regIpV4.test(line));
 					ip_ranges = [... ip_ranges, ... items];
 				})
 				.then(() => {
 					return internalIpRanges.fetchUrl(CLOUDFARE_V6_URL);
 				})
 				.then((cloudfare_data) => {
-					let items = cloudfare_data.split('\n');
+					let items = cloudfare_data.split('\n').filter((line) => regIpV6.test(line));
 					ip_ranges = [... ip_ranges, ... items];
 				})
 				.then(() => {
@@ -116,10 +119,7 @@ const internalIpRanges = {
 	 * @returns {Promise}
 	 */
 	generateConfig: (ip_ranges) => {
-		let renderEngine = new Liquid({
+		const renderEngine = utils.getRenderEngine();
-			root: __dirname + '/../templates/'
-		});
-
 		return new Promise((resolve, reject) => {
 			let template = null;
 			let filename = '/etc/nginx/conf.d/include/ip_ranges.conf';

backend/internal/nginx.js

@@ -1,10 +1,9 @@
 const _      = require('lodash');
 const fs     = require('fs');
 const logger = require('../logger').nginx;
+const config = require('../lib/config');
 const utils  = require('../lib/utils');
 const error  = require('../lib/error');
-const { Liquid } = require('liquidjs');
-const debug_mode = process.env.NODE_ENV !== 'production' || !!process.env.DEBUG;
 
 const internalNginx = {
 
@@ -29,7 +28,9 @@ const internalNginx = {
 			.then(() => {
 				// Nginx is OK
 				// We're deleting this config regardless.
-				return internalNginx.deleteConfig(host_type, host); // Don't throw errors, as the file may not exist at all
+				// Don't throw errors, as the file may not exist at all
+				// Delete the .err file too
+				return internalNginx.deleteConfig(host_type, host, false, true);
 			})
 			.then(() => {
 				return internalNginx.generateConfig(host_type, host);
@@ -64,7 +65,7 @@ const internalNginx = {
 							}
 						});
 
-						if (debug_mode) {
+						if (config.debug()) {
 							logger.error('Nginx test failed:', valid_lines.join('\n'));
 						}
 
@@ -80,6 +81,9 @@ const internalNginx = {
 							.patch({
 								meta: combined_meta
 							})
+							.then(() => {
+								internalNginx.renameConfigAsError(host_type, host);
+							})
 							.then(() => {
 								return internalNginx.deleteConfig(host_type, host, true);
 							});
@@ -97,7 +101,7 @@ const internalNginx = {
 	 * @returns {Promise}
 	 */
 	test: () => {
-		if (debug_mode) {
+		if (config.debug()) {
 			logger.info('Testing Nginx configuration');
 		}
 
@@ -121,13 +125,10 @@ const internalNginx = {
 	 * @returns {String}
 	 */
 	getConfigName: (host_type, host_id) => {
-		host_type = host_type.replace(new RegExp('-', 'g'), '_');
-
 		if (host_type === 'default') {
 			return '/data/nginx/default_host/site.conf';
 		}
-
+		return '/data/nginx/' + internalNginx.getFileFriendlyHostType(host_type) + '/' + host_id + '.conf';
-		return '/data/nginx/' + host_type + '/' + host_id + '.conf';
 	},
 
 	/**
@@ -146,12 +147,16 @@ const internalNginx = {
 				return;
 			}
 
-			let renderer          = new Liquid();
+			const renderEngine    = utils.getRenderEngine();
 			let renderedLocations = '';
 
 			const locationRendering = async () => {
 				for (let i = 0; i < host.locations.length; i++) {
-					let locationCopy = Object.assign({}, host.locations[i]);
+					let locationCopy = Object.assign({}, {access_list_id: host.access_list_id}, {certificate_id: host.certificate_id},
+						{ssl_forced: host.ssl_forced}, {caching_enabled: host.caching_enabled}, {block_exploits: host.block_exploits},
+						{allow_websocket_upgrade: host.allow_websocket_upgrade}, {http2_support: host.http2_support},
+						{hsts_enabled: host.hsts_enabled}, {hsts_subdomains: host.hsts_subdomains}, {access_list: host.access_list},
+						{certificate: host.certificate}, host.locations[i]);
 
 					if (locationCopy.forward_host.indexOf('/') > -1) {
 						const splitted = locationCopy.forward_host.split('/');
@@ -161,11 +166,13 @@ const internalNginx = {
 					}
 
 					// eslint-disable-next-line
-					renderedLocations += await renderer.parseAndRender(template, locationCopy);
+					renderedLocations += await renderEngine.parseAndRender(template, locationCopy);
 				}
+
 			};
 
 			locationRendering().then(() => resolve(renderedLocations));
+
 		});
 	},
 
@@ -175,22 +182,20 @@ const internalNginx = {
 	 * @returns {Promise}
 	 */
 	generateConfig: (host_type, host) => {
-		host_type = host_type.replace(new RegExp('-', 'g'), '_');
+		const nice_host_type = internalNginx.getFileFriendlyHostType(host_type);
 
-		if (debug_mode) {
+		if (config.debug()) {
-			logger.info('Generating ' + host_type + ' Config:', host);
+			logger.info('Generating ' + nice_host_type + ' Config:', JSON.stringify(host, null, 2));
 		}
 
-		let renderEngine = new Liquid({
+		const renderEngine = utils.getRenderEngine();
-			root: __dirname + '/../templates/'
-		});
 
 		return new Promise((resolve, reject) => {
 			let template = null;
-			let filename = internalNginx.getConfigName(host_type, host.id);
+			let filename = internalNginx.getConfigName(nice_host_type, host.id);
 
 			try {
-				template = fs.readFileSync(__dirname + '/../templates/' + host_type + '.conf', {encoding: 'utf8'});
+				template = fs.readFileSync(__dirname + '/../templates/' + nice_host_type + '.conf', {encoding: 'utf8'});
 			} catch (err) {
 				reject(new error.ConfigurationError(err.message));
 				return;
@@ -200,7 +205,7 @@ const internalNginx = {
 			let origLocations;
 
 			// Manipulate the data a bit before sending it to the template
-			if (host_type !== 'default') {
+			if (nice_host_type !== 'default') {
 				host.use_default_location = true;
 				if (typeof host.advanced_config !== 'undefined' && host.advanced_config) {
 					host.use_default_location = !internalNginx.advancedConfigHasDefaultLocation(host.advanced_config);
@@ -208,6 +213,7 @@ const internalNginx = {
 			}
 
 			if (host.locations) {
+				//logger.info ('host.locations = ' + JSON.stringify(host.locations, null, 2));
 				origLocations    = [].concat(host.locations);
 				locationsPromise = internalNginx.renderLocations(host).then((renderedLocations) => {
 					host.locations = renderedLocations;
@@ -224,13 +230,16 @@ const internalNginx = {
 				locationsPromise = Promise.resolve();
 			}
 
+			// Set the IPv6 setting for the host
+			host.ipv6 = internalNginx.ipv6Enabled();
+
 			locationsPromise.then(() => {
 				renderEngine
 					.parseAndRender(template, host)
 					.then((config_text) => {
 						fs.writeFileSync(filename, config_text, {encoding: 'utf8'});
 
-						if (debug_mode) {
+						if (config.debug()) {
 							logger.success('Wrote config:', filename, config_text);
 						}
 
@@ -240,7 +249,7 @@ const internalNginx = {
 						resolve(true);
 					})
 					.catch((err) => {
-						if (debug_mode) {
+						if (config.debug()) {
 							logger.warn('Could not write ' + filename + ':', err.message);
 						}
 
@@ -259,17 +268,16 @@ const internalNginx = {
 	 * @returns {Promise}
 	 */
 	generateLetsEncryptRequestConfig: (certificate) => {
-		if (debug_mode) {
+		if (config.debug()) {
 			logger.info('Generating LetsEncrypt Request Config:', certificate);
 		}
 
-		let renderEngine = new Liquid({
+		const renderEngine = utils.getRenderEngine();
-			root: __dirname + '/../templates/'
-		});
 
 		return new Promise((resolve, reject) => {
 			let template = null;
 			let filename = '/data/nginx/temp/letsencrypt_' + certificate.id + '.conf';
+
 			try {
 				template = fs.readFileSync(__dirname + '/../templates/letsencrypt-request.conf', {encoding: 'utf8'});
 			} catch (err) {
@@ -277,19 +285,21 @@ const internalNginx = {
 				return;
 			}
 
+			certificate.ipv6 = internalNginx.ipv6Enabled();
+
 			renderEngine
 				.parseAndRender(template, certificate)
 				.then((config_text) => {
 					fs.writeFileSync(filename, config_text, {encoding: 'utf8'});
 
-					if (debug_mode) {
+					if (config.debug()) {
 						logger.success('Wrote config:', filename, config_text);
 					}
 
 					resolve(true);
 				})
 				.catch((err) => {
-					if (debug_mode) {
+					if (config.debug()) {
 						logger.warn('Could not write ' + filename + ':', err.message);
 					}
 
@@ -298,33 +308,39 @@ const internalNginx = {
 		});
 	},
 
+	/**
+	 * A simple wrapper around unlinkSync that writes to the logger
+	 *
+	 * @param   {String}  filename
+	 */
+	deleteFile: (filename) => {
+		logger.debug('Deleting file: ' + filename);
+		try {
+			fs.unlinkSync(filename);
+		} catch (err) {
+			logger.debug('Could not delete file:', JSON.stringify(err, null, 2));
+		}
+	},
+
+	/**
+	 *
+	 * @param   {String} host_type
+	 * @returns String
+	 */
+	getFileFriendlyHostType: (host_type) => {
+		return host_type.replace(new RegExp('-', 'g'), '_');
+	},
+
 	/**
 	 * This removes the temporary nginx config file generated by `generateLetsEncryptRequestConfig`
 	 *
 	 * @param   {Object}  certificate
-	 * @param   {Boolean} [throw_errors]
 	 * @returns {Promise}
 	 */
-	deleteLetsEncryptRequestConfig: (certificate, throw_errors) => {
+	deleteLetsEncryptRequestConfig: (certificate) => {
-		return new Promise((resolve, reject) => {
+		const config_file = '/data/nginx/temp/letsencrypt_' + certificate.id + '.conf';
-			try {
+		return new Promise((resolve/*, reject*/) => {
-				let config_file = '/data/nginx/temp/letsencrypt_' + certificate.id + '.conf';
+			internalNginx.deleteFile(config_file);
-
-				if (debug_mode) {
-					logger.warn('Deleting nginx config: ' + config_file);
-				}
-
-				fs.unlinkSync(config_file);
-			} catch (err) {
-				if (debug_mode) {
-					logger.warn('Could not delete config:', err.message);
-				}
-
-				if (throw_errors) {
-					reject(err);
-				}
-			}
-
 			resolve();
 		});
 	},
@@ -332,35 +348,42 @@ const internalNginx = {
 	/**
 	 * @param   {String}  host_type
 	 * @param   {Object}  [host]
-	 * @param   {Boolean} [throw_errors]
+	 * @param   {Boolean} [delete_err_file]
 	 * @returns {Promise}
 	 */
-	deleteConfig: (host_type, host, throw_errors) => {
+	deleteConfig: (host_type, host, delete_err_file) => {
-		host_type = host_type.replace(new RegExp('-', 'g'), '_');
+		const config_file     = internalNginx.getConfigName(internalNginx.getFileFriendlyHostType(host_type), typeof host === 'undefined' ? 0 : host.id);
+		const config_file_err = config_file + '.err';
 
-		return new Promise((resolve, reject) => {
+		return new Promise((resolve/*, reject*/) => {
-			try {
+			internalNginx.deleteFile(config_file);
-				let config_file = internalNginx.getConfigName(host_type, typeof host === 'undefined' ? 0 : host.id);
+			if (delete_err_file) {
-
+				internalNginx.deleteFile(config_file_err);
-				if (debug_mode) {
-					logger.warn('Deleting nginx config: ' + config_file);
 			}
-
-				fs.unlinkSync(config_file);
-			} catch (err) {
-				if (debug_mode) {
-					logger.warn('Could not delete config:', err.message);
-				}
-
-				if (throw_errors) {
-					reject(err);
-				}
-			}
-
 			resolve();
 		});
 	},
 
+	/**
+	 * @param   {String}  host_type
+	 * @param   {Object}  [host]
+	 * @returns {Promise}
+	 */
+	renameConfigAsError: (host_type, host) => {
+		const config_file     = internalNginx.getConfigName(internalNginx.getFileFriendlyHostType(host_type), typeof host === 'undefined' ? 0 : host.id);
+		const config_file_err = config_file + '.err';
+
+		return new Promise((resolve/*, reject*/) => {
+			fs.unlink(config_file, () => {
+				// ignore result, continue
+				fs.rename(config_file, config_file_err, () => {
+					// also ignore result, as this is a debugging informative file anyway
+					resolve();
+				});
+			});
+		});
+	},
+
 	/**
 	 * @param   {String}  host_type
 	 * @param   {Array}   hosts
@@ -378,13 +401,12 @@ const internalNginx = {
 	/**
 	 * @param   {String}  host_type
 	 * @param   {Array}   hosts
-	 * @param   {Boolean} [throw_errors]
 	 * @returns {Promise}
 	 */
-	bulkDeleteConfigs: (host_type, hosts, throw_errors) => {
+	bulkDeleteConfigs: (host_type, hosts) => {
 		let promises = [];
 		hosts.map(function (host) {
-			promises.push(internalNginx.deleteConfig(host_type, host, throw_errors));
+			promises.push(internalNginx.deleteConfig(host_type, host, true));
 		});
 
 		return Promise.all(promises);
@@ -394,8 +416,20 @@ const internalNginx = {
 	 * @param   {string}  config
 	 * @returns {boolean}
 	 */
-	advancedConfigHasDefaultLocation: function (config) {
+	advancedConfigHasDefaultLocation: function (cfg) {
-		return !!config.match(/^(?:.*;)?\s*?location\s*?\/\s*?{/im);
+		return !!cfg.match(/^(?:.*;)?\s*?location\s*?\/\s*?{/im);
+	},
+
+	/**
+	 * @returns {boolean}
+	 */
+	ipv6Enabled: function () {
+		if (typeof process.env.DISABLE_IPV6 !== 'undefined') {
+			const disabled = process.env.DISABLE_IPV6.toLowerCase();
+			return !(disabled === 'on' || disabled === 'true' || disabled === '1' || disabled === 'yes');
+		}
+
+		return true;
 	}
 };
 

backend/internal/proxy-host.js

@@ -1,5 +1,6 @@
 const _                   = require('lodash');
 const error               = require('../lib/error');
+const utils               = require('../lib/utils');
 const proxyHostModel      = require('../models/proxy_host');
 const internalHost        = require('./host');
 const internalNginx       = require('./nginx');
@@ -7,7 +8,7 @@ const internalAuditLog    = require('./audit-log');
 const internalCertificate = require('./certificate');
 
 function omissions () {
-	return ['is_deleted'];
+	return ['is_deleted', 'owner.is_deleted'];
 }
 
 const internalProxyHost = {
@@ -47,10 +48,16 @@ const internalProxyHost = {
 				data.owner_user_id = access.token.getUserId(1);
 				data               = internalHost.cleanSslHstsData(data);
 
+				// Fix for db field not having a default value
+				// for this optional field.
+				if (typeof data.advanced_config === 'undefined') {
+					data.advanced_config = '';
+				}
+
 				return proxyHostModel
 					.query()
-					.omit(omissions())
+					.insertAndFetch(data)
-					.insertAndFetch(data);
+					.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
 				if (create_certificate) {
@@ -73,7 +80,7 @@ const internalProxyHost = {
 				// re-fetch with cert
 				return internalProxyHost.get(access, {
 					id:     row.id,
-					expand: ['certificate', 'owner', 'access_list']
+					expand: ['certificate', 'owner', 'access_list.[clients,items]']
 				});
 			})
 			.then((row) => {
@@ -170,6 +177,7 @@ const internalProxyHost = {
 					.query()
 					.where({id: data.id})
 					.patch(data)
+					.then(utils.omitRow(omissions()))
 					.then((saved_row) => {
 						// Add to audit log
 						return internalAuditLog.add(access, {
@@ -179,16 +187,20 @@ const internalProxyHost = {
 							meta:        data
 						})
 							.then(() => {
-								return _.omit(saved_row, omissions());
+								return saved_row;
 							});
 					});
 			})
 			.then(() => {
 				return internalProxyHost.get(access, {
 					id:     data.id,
-					expand: ['owner', 'certificate', 'access_list']
+					expand: ['owner', 'certificate', 'access_list.[clients,items]']
 				})
 					.then((row) => {
+						if (!row.enabled) {
+							// No need to add nginx config if host is disabled
+							return row;
+						}
 						// Configure nginx
 						return internalNginx.configure(proxyHostModel, 'proxy_host', row)
 							.then((new_meta) => {
@@ -219,31 +231,29 @@ const internalProxyHost = {
 					.query()
 					.where('is_deleted', 0)
 					.andWhere('id', data.id)
-					.allowEager('[owner,access_list,certificate]')
+					.allowGraph('[owner,access_list.[clients,items],certificate]')
 					.first();
 
 				if (access_data.permission_visibility !== 'all') {
 					query.andWhere('owner_user_id', access.token.getUserId(1));
 				}
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
-					row = internalHost.cleanRowCertificateMeta(row);
-					return _.omit(row, omissions());
-				} else {
 					throw new error.ItemNotFoundError(data.id);
 				}
+				row = internalHost.cleanRowCertificateMeta(row);
+				// Custom omissions
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
+					row = _.omit(row, data.omit);
+				}
+				return row;
 			});
 	},
 
@@ -260,7 +270,7 @@ const internalProxyHost = {
 				return internalProxyHost.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				}
 
@@ -308,7 +318,7 @@ const internalProxyHost = {
 				});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (row.enabled) {
 					throw new error.ValidationError('Host is already enabled');
@@ -354,7 +364,7 @@ const internalProxyHost = {
 				return internalProxyHost.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (!row.enabled) {
 					throw new error.ValidationError('Host is already disabled');
@@ -405,8 +415,7 @@ const internalProxyHost = {
 					.query()
 					.where('is_deleted', 0)
 					.groupBy('id')
-					.omit(['is_deleted'])
+					.allowGraph('[owner,access_list,certificate]')
-					.allowEager('[owner,access_list,certificate]')
 					.orderBy('domain_names', 'ASC');
 
 				if (access_data.permission_visibility !== 'all') {
@@ -421,10 +430,10 @@ const internalProxyHost = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			})
 			.then((rows) => {
 				if (typeof expand !== 'undefined' && expand !== null && expand.indexOf('certificate') !== -1) {

backend/internal/redirection-host.js

@@ -1,5 +1,6 @@
 const _                    = require('lodash');
 const error                = require('../lib/error');
+const utils                = require('../lib/utils');
 const redirectionHostModel = require('../models/redirection_host');
 const internalHost         = require('./host');
 const internalNginx        = require('./nginx');
@@ -47,10 +48,16 @@ const internalRedirectionHost = {
 				data.owner_user_id = access.token.getUserId(1);
 				data               = internalHost.cleanSslHstsData(data);
 
+				// Fix for db field not having a default value
+				// for this optional field.
+				if (typeof data.advanced_config === 'undefined') {
+					data.advanced_config = '';
+				}
+
 				return redirectionHostModel
 					.query()
-					.omit(omissions())
+					.insertAndFetch(data)
-					.insertAndFetch(data);
+					.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
 				if (create_certificate) {
@@ -65,9 +72,8 @@ const internalRedirectionHost = {
 						.then(() => {
 							return row;
 						});
-				} else {
-					return row;
 				}
+				return row;
 			})
 			.then((row) => {
 				// re-fetch with cert
@@ -218,31 +224,29 @@ const internalRedirectionHost = {
 					.query()
 					.where('is_deleted', 0)
 					.andWhere('id', data.id)
-					.allowEager('[owner,certificate]')
+					.allowGraph('[owner,certificate]')
 					.first();
 
 				if (access_data.permission_visibility !== 'all') {
 					query.andWhere('owner_user_id', access.token.getUserId(1));
 				}
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
-					row = internalHost.cleanRowCertificateMeta(row);
-					return _.omit(row, omissions());
-				} else {
 					throw new error.ItemNotFoundError(data.id);
 				}
+				row = internalHost.cleanRowCertificateMeta(row);
+				// Custom omissions
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
+					row = _.omit(row, data.omit);
+				}
+				return row;
 			});
 	},
 
@@ -259,7 +263,7 @@ const internalRedirectionHost = {
 				return internalRedirectionHost.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				}
 
@@ -307,7 +311,7 @@ const internalRedirectionHost = {
 				});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (row.enabled) {
 					throw new error.ValidationError('Host is already enabled');
@@ -353,7 +357,7 @@ const internalRedirectionHost = {
 				return internalRedirectionHost.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (!row.enabled) {
 					throw new error.ValidationError('Host is already disabled');
@@ -404,8 +408,7 @@ const internalRedirectionHost = {
 					.query()
 					.where('is_deleted', 0)
 					.groupBy('id')
-					.omit(['is_deleted'])
+					.allowGraph('[owner,certificate]')
-					.allowEager('[owner,certificate]')
 					.orderBy('domain_names', 'ASC');
 
 				if (access_data.permission_visibility !== 'all') {
@@ -420,10 +423,10 @@ const internalRedirectionHost = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			})
 			.then((rows) => {
 				if (typeof expand !== 'undefined' && expand !== null && expand.indexOf('certificate') !== -1) {

backend/internal/stream.js

@@ -1,5 +1,6 @@
 const _                = require('lodash');
 const error            = require('../lib/error');
+const utils            = require('../lib/utils');
 const streamModel      = require('../models/stream');
 const internalNginx    = require('./nginx');
 const internalAuditLog = require('./audit-log');
@@ -27,8 +28,8 @@ const internalStream = {
 
 				return streamModel
 					.query()
-					.omit(omissions())
+					.insertAndFetch(data)
-					.insertAndFetch(data);
+					.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
 				// Configure nginx
@@ -71,8 +72,8 @@ const internalStream = {
 
 				return streamModel
 					.query()
-					.omit(omissions())
 					.patchAndFetchById(row.id, data)
+					.then(utils.omitRow(omissions()))
 					.then((saved_row) => {
 						return internalNginx.configure(streamModel, 'stream', saved_row)
 							.then(() => {
@@ -88,7 +89,7 @@ const internalStream = {
 							meta:        data
 						})
 							.then(() => {
-								return _.omit(saved_row, omissions());
+								return saved_row;
 							});
 					});
 			});
@@ -113,30 +114,28 @@ const internalStream = {
 					.query()
 					.where('is_deleted', 0)
 					.andWhere('id', data.id)
-					.allowEager('[owner]')
+					.allowGraph('[owner]')
 					.first();
 
 				if (access_data.permission_visibility !== 'all') {
 					query.andWhere('owner_user_id', access.token.getUserId(1));
 				}
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
-					return _.omit(row, omissions());
-				} else {
 					throw new error.ItemNotFoundError(data.id);
 				}
+				// Custom omissions
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
+					row = _.omit(row, data.omit);
+				}
+				return row;
 			});
 	},
 
@@ -153,7 +152,7 @@ const internalStream = {
 				return internalStream.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				}
 
@@ -201,7 +200,7 @@ const internalStream = {
 				});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (row.enabled) {
 					throw new error.ValidationError('Host is already enabled');
@@ -247,7 +246,7 @@ const internalStream = {
 				return internalStream.get(access, {id: data.id});
 			})
 			.then((row) => {
-				if (!row) {
+				if (!row || !row.id) {
 					throw new error.ItemNotFoundError(data.id);
 				} else if (!row.enabled) {
 					throw new error.ValidationError('Host is already disabled');
@@ -298,8 +297,7 @@ const internalStream = {
 					.query()
 					.where('is_deleted', 0)
 					.groupBy('id')
-					.omit(['is_deleted'])
+					.allowGraph('[owner]')
-					.allowEager('[owner]')
 					.orderBy('incoming_port', 'ASC');
 
 				if (access_data.permission_visibility !== 'all') {
@@ -314,10 +312,10 @@ const internalStream = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			});
 	},
 

backend/internal/token.js

@@ -24,7 +24,7 @@ module.exports = {
 
 		return userModel
 			.query()
-			.where('email', data.identity)
+			.where('email', data.identity.toLowerCase().trim())
 			.andWhere('is_deleted', 0)
 			.andWhere('is_disabled', 0)
 			.first()

backend/internal/user.js

@@ -1,5 +1,6 @@
 const _                   = require('lodash');
 const error               = require('../lib/error');
+const utils               = require('../lib/utils');
 const userModel           = require('../models/user');
 const userPermissionModel = require('../models/user_permission');
 const authModel           = require('../models/auth');
@@ -35,8 +36,8 @@ const internalUser = {
 
 				return userModel
 					.query()
-					.omit(omissions())
+					.insertAndFetch(data)
-					.insertAndFetch(data);
+					.then(utils.omitRow(omissions()));
 			})
 			.then((user) => {
 				if (auth) {
@@ -140,11 +141,8 @@ const internalUser = {
 
 				return userModel
 					.query()
-					.omit(omissions())
 					.patchAndFetchById(user.id, data)
-					.then((saved_user) => {
+					.then(utils.omitRow(omissions()));
-						return _.omit(saved_user, omissions());
-					});
 			})
 			.then(() => {
 				return internalUser.get(access, {id: data.id});
@@ -186,26 +184,24 @@ const internalUser = {
 					.query()
 					.where('is_deleted', 0)
 					.andWhere('id', data.id)
-					.allowEager('[permissions]')
+					.allowGraph('[permissions]')
 					.first();
 
-				// Custom omissions
-				if (typeof data.omit !== 'undefined' && data.omit !== null) {
-					query.omit(data.omit);
-				}
-
 				if (typeof data.expand !== 'undefined' && data.expand !== null) {
-					query.eager('[' + data.expand.join(', ') + ']');
+					query.withGraphFetched('[' + data.expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRow(omissions()));
 			})
 			.then((row) => {
-				if (row) {
+				if (!row || !row.id) {
-					return _.omit(row, omissions());
-				} else {
 					throw new error.ItemNotFoundError(data.id);
 				}
+				// Custom omissions
+				if (typeof data.omit !== 'undefined' && data.omit !== null) {
+					row = _.omit(row, data.omit);
+				}
+				return row;
 			});
 	},
 
@@ -322,8 +318,7 @@ const internalUser = {
 					.query()
 					.where('is_deleted', 0)
 					.groupBy('id')
-					.omit(['is_deleted'])
+					.allowGraph('[permissions]')
-					.allowEager('[permissions]')
 					.orderBy('name', 'ASC');
 
 				// Query is used for searching
@@ -335,10 +330,10 @@ const internalUser = {
 				}
 
 				if (typeof expand !== 'undefined' && expand !== null) {
-					query.eager('[' + expand.join(', ') + ']');
+					query.withGraphFetched('[' + expand.join(', ') + ']');
 				}
 
-				return query;
+				return query.then(utils.omitRows(omissions()));
 			});
 	},
 

backend/knexfile.js

@@ -1,6 +1,6 @@
 module.exports = {
 	development: {
-		client:     'mysql',
+		client:     'mysql2',
 		migrations: {
 			tableName: 'migrations',
 			stub:      'lib/migrate_template.js',
@@ -9,7 +9,7 @@ module.exports = {
 	},
 
 	production: {
-		client:     'mysql',
+		client:     'mysql2',
 		migrations: {
 			tableName: 'migrations',
 			stub:      'lib/migrate_template.js',

backend/lib/access.js

@@ -10,7 +10,7 @@
 
 const _              = require('lodash');
 const logger         = require('../logger').access;
-const validator      = require('ajv');
+const Ajv            = require('ajv/dist/2020');
 const error          = require('./error');
 const userModel      = require('../models/user');
 const proxyHostModel = require('../models/proxy_host');
@@ -55,8 +55,8 @@ module.exports = function (token_string) {
 								.where('id', token_data.attrs.id)
 								.andWhere('is_deleted', 0)
 								.andWhere('is_disabled', 0)
-								.allowEager('[permissions]')
+								.allowGraph('[permissions]')
-								.eager('[permissions]')
+								.withGraphFetched('[permissions]')
 								.first()
 								.then((user) => {
 									if (user) {
@@ -174,7 +174,6 @@ module.exports = function (token_string) {
 
 		let schema = {
 			$id:                  'objects',
-			$schema:              'http://json-schema.org/draft-07/schema#',
 			description:          'Actor Properties',
 			type:                 'object',
 			additionalProperties: false,
@@ -251,7 +250,7 @@ module.exports = function (token_string) {
 						// Initialised, token decoded ok
 						return this.getObjectSchema(permission)
 							.then((objectSchema) => {
-								let data_schema = {
+								const data_schema = {
 									[permission]: {
 										data:                         data,
 										scope:                        Token.get('scope'),
@@ -267,24 +266,18 @@ module.exports = function (token_string) {
 								};
 
 								let permissionSchema = {
-									$schema:              'http://json-schema.org/draft-07/schema#',
 									$async:               true,
 									$id:                  'permissions',
+									type:                 'object',
 									additionalProperties: false,
 									properties:           {}
 								};
 
 								permissionSchema.properties[permission] = require('./access/' + permission.replace(/:/gim, '-') + '.json');
 
-								// logger.info('objectSchema', JSON.stringify(objectSchema, null, 2));
+								const ajv = new Ajv({
-								// logger.info('permissionSchema', JSON.stringify(permissionSchema, null, 2));
-								// logger.info('data_schema', JSON.stringify(data_schema, null, 2));
-
-								let ajv = validator({
 									verbose:      true,
 									allErrors:    true,
-									format:       'full',
-									missingRefs:  'fail',
 									breakOnError: true,
 									coerceTypes:  true,
 									schemas:      [

backend/lib/access/permissions.json

@@ -1,5 +1,4 @@
 {
-	"$schema": "http://json-schema.org/draft-07/schema#",
 	"$id": "perms",
 	"definitions": {
 		"view": {

backend/lib/access/roles.json

@@ -1,5 +1,4 @@
 {
-	"$schema": "http://json-schema.org/draft-07/schema#",
 	"$id": "roles",
 	"definitions": {
 		"admin": {

backend/lib/certbot.js

@@ -0,0 +1,78 @@
+const dnsPlugins = require('../global/certbot-dns-plugins.json');
+const utils      = require('./utils');
+const error      = require('./error');
+const logger     = require('../logger').certbot;
+const batchflow  = require('batchflow');
+
+const CERTBOT_VERSION_REPLACEMENT = '$(certbot --version | grep -Eo \'[0-9](\\.[0-9]+)+\')';
+
+const certbot = {
+
+	/**
+	 * @param {array} pluginKeys
+	 */
+	installPlugins: async function (pluginKeys) {
+		let hasErrors = false;
+
+		return new Promise((resolve, reject) => {
+			if (pluginKeys.length === 0) {
+				resolve();
+				return;
+			}
+
+			batchflow(pluginKeys).sequential()
+				.each((i, pluginKey, next) => {
+					certbot.installPlugin(pluginKey)
+						.then(() => {
+							next();
+						})
+						.catch((err) => {
+							hasErrors = true;
+							next(err);
+						});
+				})
+				.error((err) => {
+					logger.error(err.message);
+				})
+				.end(() => {
+					if (hasErrors) {
+						reject(new error.CommandError('Some plugins failed to install. Please check the logs above', 1));
+					} else {
+						resolve();
+					}
+				});
+		});
+	},
+
+	/**
+	 * Installs a cerbot plugin given the key for the object from
+	 * ../global/certbot-dns-plugins.json
+	 *
+	 * @param   {string}  pluginKey
+	 * @returns {Object}
+	 */
+	installPlugin: async function (pluginKey) {
+		if (typeof dnsPlugins[pluginKey] === 'undefined') {
+			// throw Error(`Certbot plugin ${pluginKey} not found`);
+			throw new error.ItemNotFoundError(pluginKey);
+		}
+
+		const plugin = dnsPlugins[pluginKey];
+		logger.start(`Installing ${pluginKey}...`);
+
+		plugin.version      = plugin.version.replace(/{{certbot-version}}/g, CERTBOT_VERSION_REPLACEMENT);
+		plugin.dependencies = plugin.dependencies.replace(/{{certbot-version}}/g, CERTBOT_VERSION_REPLACEMENT);
+
+		const cmd = '. /opt/certbot/bin/activate && pip install --no-cache-dir ' + plugin.dependencies + ' ' + plugin.package_name + plugin.version + ' ' + ' && deactivate';
+		return utils.exec(cmd)
+			.then((result) => {
+				logger.complete(`Installed ${pluginKey}`);
+				return result;
+			})
+			.catch((err) => {
+				throw err;
+			});
+	},
+};
+
+module.exports = certbot;

backend/lib/config.js

@@ -0,0 +1,203 @@
+const fs      = require('fs');
+const NodeRSA = require('node-rsa');
+const logger  = require('../logger').global;
+
+const keysFile = '/data/keys.json';
+
+let instance = null;
+
+// 1. Load from config file first (not recommended anymore)
+// 2. Use config env variables next
+const configure = () => {
+	const filename = (process.env.NODE_CONFIG_DIR || './config') + '/' + (process.env.NODE_ENV || 'default') + '.json';
+	if (fs.existsSync(filename)) {
+		let configData;
+		try {
+			configData = require(filename);
+		} catch (err) {
+			// do nothing
+		}
+
+		if (configData && configData.database) {
+			logger.info(`Using configuration from file: ${filename}`);
+			instance      = configData;
+			instance.keys = getKeys();
+			return;
+		}
+	}
+
+	const envMysqlHost = process.env.DB_MYSQL_HOST || null;
+	const envMysqlUser = process.env.DB_MYSQL_USER || null;
+	const envMysqlName = process.env.DB_MYSQL_NAME || null;
+	if (envMysqlHost && envMysqlUser && envMysqlName) {
+		// we have enough mysql creds to go with mysql
+		logger.info('Using MySQL configuration');
+		instance = {
+			database: {
+				engine:   'mysql2',
+				host:     envMysqlHost,
+				port:     process.env.DB_MYSQL_PORT || 3306,
+				user:     envMysqlUser,
+				password: process.env.DB_MYSQL_PASSWORD,
+				name:     envMysqlName,
+			},
+			keys: getKeys(),
+		};
+		return;
+	}
+	const envPostgresqlHost = process.env.DB_POSTGRESQL_HOST || null;
+	const envPostgresqlUser = process.env.DB_POSTGRESQL_USER || null;
+	const envPostgresqlName = process.env.DB_POSTGRESQL_NAME || null;
+	if (envPostgresqlHost && envPostgresqlUser && envPostgresqlName) {
+		// we have enough mysql creds to go with mysql
+		logger.info('Using POSTGRESQL configuration');
+		instance = {
+			database: {
+				engine:   'pg',
+				host:     envPostgresqlHost,
+				port:     process.env.DB_POSTGRESQL_PORT || 3306,
+				user:     envPostgresqlUser,
+				password: process.env.DB_POSTGRESQL_PASSWORD,
+				name:     envPostgresqlName,
+			},
+			keys: getKeys(),
+		};
+		return;
+	}
+
+	const envSqliteFile = process.env.DB_SQLITE_FILE || '/data/database.sqlite';
+	logger.info(`Using Sqlite: ${envSqliteFile}`);
+	instance = {
+		database: {
+			engine: 'knex-native',
+			knex:   {
+				client:     'sqlite3',
+				connection: {
+					filename: envSqliteFile
+				},
+				useNullAsDefault: true
+			}
+		},
+		keys: getKeys(),
+	};
+};
+
+const getKeys = () => {
+	// Get keys from file
+	if (!fs.existsSync(keysFile)) {
+		generateKeys();
+	} else if (process.env.DEBUG) {
+		logger.info('Keys file exists OK');
+	}
+	try {
+		return require(keysFile);
+	} catch (err) {
+		logger.error('Could not read JWT key pair from config file: ' + keysFile, err);
+		process.exit(1);
+	}
+};
+
+const generateKeys = () => {
+	logger.info('Creating a new JWT key pair...');
+	// Now create the keys and save them in the config.
+	const key = new NodeRSA({ b: 2048 });
+	key.generateKeyPair();
+
+	const keys = {
+		key: key.exportKey('private').toString(),
+		pub: key.exportKey('public').toString(),
+	};
+
+	// Write keys config
+	try {
+		fs.writeFileSync(keysFile, JSON.stringify(keys, null, 2));
+	} catch (err) {
+		logger.error('Could not write JWT key pair to config file: ' + keysFile + ': ' + err.message);
+		process.exit(1);
+	}
+	logger.info('Wrote JWT key pair to config file: ' + keysFile);
+};
+
+module.exports = {
+
+	/**
+	 *
+	 * @param   {string}  key   ie: 'database' or 'database.engine'
+	 * @returns {boolean}
+	 */
+	has: function(key) {
+		instance === null && configure();
+		const keys = key.split('.');
+		let level  = instance;
+		let has    = true;
+		keys.forEach((keyItem) =>{
+			if (typeof level[keyItem] === 'undefined') {
+				has = false;
+			} else {
+				level = level[keyItem];
+			}
+		});
+
+		return has;
+	},
+
+	/**
+	 * Gets a specific key from the top level
+	 *
+	 * @param {string} key
+	 * @returns {*}
+	 */
+	get: function (key) {
+		instance === null && configure();
+		if (key && typeof instance[key] !== 'undefined') {
+			return instance[key];
+		}
+		return instance;
+	},
+
+	/**
+	 * Is this a sqlite configuration?
+	 *
+	 * @returns {boolean}
+	 */
+	isSqlite: function () {
+		instance === null && configure();
+		return instance.database.knex && instance.database.knex.client === 'sqlite3';
+	},
+
+	/**
+	 * Are we running in debug mdoe?
+	 *
+	 * @returns {boolean}
+	 */
+	debug: function () {
+		return !!process.env.DEBUG;
+	},
+
+	/**
+	 * Returns a public key
+	 *
+	 * @returns {string}
+	 */
+	getPublicKey: function () {
+		instance === null && configure();
+		return instance.keys.pub;
+	},
+
+	/**
+	 * Returns a private key
+	 *
+	 * @returns {string}
+	 */
+	getPrivateKey: function () {
+		instance === null && configure();
+		return instance.keys.key;
+	},
+
+	/**
+	 * @returns {boolean}
+	 */
+	useLetsencryptStaging: function () {
+		return !!process.env.LE_STAGING;
+	}
+};

backend/lib/error.js

@@ -82,7 +82,16 @@ module.exports = {
 		this.message  = message;
 		this.public   = false;
 		this.status   = 400;
-	}
+	},
+
+	CommandError: function (stdErr, code, previous) {
+		Error.captureStackTrace(this, this.constructor);
+		this.name     = this.constructor.name;
+		this.previous = previous;
+		this.message  = stdErr;
+		this.code     = code;
+		this.public   = false;
+	},
 };
 
 _.forEach(module.exports, function (error) {

backend/lib/express/cors.js

@@ -1,15 +1,5 @@
-const validator = require('../validator');
-
 module.exports = function (req, res, next) {
-
 	if (req.headers.origin) {
-
-		// very relaxed validation....
-		validator({
-			type:    'string',
-			pattern: '^[a-z\\-]+:\\/\\/(?:[\\w\\-\\.]+(:[0-9]+)?/?)?$'
-		}, req.headers.origin)
-			.then(function () {
 		res.set({
 			'Access-Control-Allow-Origin':      req.headers.origin,
 			'Access-Control-Allow-Credentials': true,
@@ -19,12 +9,8 @@ module.exports = function (req, res, next) {
 			'Access-Control-Expose-Headers':    'X-Dataset-Total, X-Dataset-Offset, X-Dataset-Limit'
 		});
 		next();
-			})
-			.catch(next);
-
 	} else {
 		// No origin
 		next();
 	}
-
 };

backend/lib/helpers.js

@@ -27,6 +27,24 @@ module.exports = {
 		}
 
 		return null;
+	},
+
+	convertIntFieldsToBool: function (obj, fields) {
+		fields.forEach(function (field) {
+			if (typeof obj[field] !== 'undefined') {
+				obj[field] = obj[field] === 1;
+			}
+		});
+		return obj;
+	},
+
+	convertBoolFieldsToInt: function (obj, fields) {
+		fields.forEach(function (field) {
+			if (typeof obj[field] !== 'undefined') {
+				obj[field] = obj[field] ? 1 : 0;
+			}
+		});
+		return obj;
 	}
 
 };

backend/lib/utils.js

@@ -1,14 +1,41 @@
+const _          = require('lodash');
 const exec       = require('child_process').exec;
+const execFile   = require('child_process').execFile;
+const { Liquid } = require('liquidjs');
+const logger     = require('../logger').global;
+const error      = require('./error');
 
 module.exports = {
 
+	exec: async function(cmd, options = {}) {
+		logger.debug('CMD:', cmd);
+
+		const { stdout, stderr } = await new Promise((resolve, reject) => {
+			const child = exec(cmd, options, (isError, stdout, stderr) => {
+				if (isError) {
+					reject(new error.CommandError(stderr, isError));
+				} else {
+					resolve({ stdout, stderr });
+				}
+			});
+
+			child.on('error', (e) => {
+				reject(new error.CommandError(stderr, 1, e));
+			});
+		});
+		return stdout;
+	},
+
 	/**
 	 * @param   {String} cmd
+	 * @param   {Array}  args
 	 * @returns {Promise}
 	 */
-	exec: function (cmd) {
+	execFile: function (cmd, args) {
+		// logger.debug('CMD: ' + cmd + ' ' + (args ? args.join(' ') : ''));
+
 		return new Promise((resolve, reject) => {
-			exec(cmd, function (err, stdout, /*stderr*/) {
+			execFile(cmd, args, function (err, stdout, /*stderr*/) {
 				if (err && typeof err === 'object') {
 					reject(err);
 				} else {
@@ -16,5 +43,64 @@ module.exports = {
 				}
 			});
 		});
+	},
+
+	/**
+	 * Used in objection query builder
+	 *
+	 * @param   {Array}  omissions
+	 * @returns {Function}
+	 */
+	omitRow: function (omissions) {
+		/**
+		 * @param   {Object} row
+		 * @returns {Object}
+		 */
+		return (row) => {
+			return _.omit(row, omissions);
+		};
+	},
+
+	/**
+	 * Used in objection query builder
+	 *
+	 * @param   {Array}  omissions
+	 * @returns {Function}
+	 */
+	omitRows: function (omissions) {
+		/**
+		 * @param   {Array} rows
+		 * @returns {Object}
+		 */
+		return (rows) => {
+			rows.forEach((row, idx) => {
+				rows[idx] = _.omit(row, omissions);
+			});
+			return rows;
+		};
+	},
+
+	/**
+	 * @returns {Object} Liquid render engine
+	 */
+	getRenderEngine: function () {
+		const renderEngine = new Liquid({
+			root: __dirname + '/../templates/'
+		});
+
+		/**
+		 * nginxAccessRule expects the object given to have 2 properties:
+		 *
+		 * directive  string
+		 * address    string
+		 */
+		renderEngine.registerFilter('nginxAccessRule', (v) => {
+			if (typeof v.directive !== 'undefined' && typeof v.address !== 'undefined' && v.directive && v.address) {
+				return `${v.directive} ${v.address};`;
+			}
+			return '';
+		});
+
+		return renderEngine;
 	}
 };

backend/lib/validator/api.js

@@ -1,13 +1,12 @@
+const Ajv   = require('ajv/dist/2020');
 const error = require('../error');
-const path   = require('path');
-const parser = require('json-schema-ref-parser');
 
-const ajv = require('ajv')({
+const ajv = new Ajv({
 	verbose:         true,
-	validateSchema: true,
+	allErrors:       true,
-	allErrors:      false,
+	allowUnionTypes: true,
-	format:         'full',
+	strict:          false,
-	coerceTypes:    true
+	coerceTypes:     true,
 });
 
 /**
@@ -17,12 +16,18 @@ const ajv = require('ajv')({
  */
 function apiValidator (schema, payload/*, description*/) {
 	return new Promise(function Promise_apiValidator (resolve, reject) {
-		if (typeof payload === 'undefined') {
+		if (schema === null) {
-			reject(new error.ValidationError('Payload is undefined'));
+			reject(new error.ValidationError('Schema is undefined'));
+			return;
 		}
 
-		let validate = ajv.compile(schema);
+		if (typeof payload === 'undefined') {
-		let valid    = validate(payload);
+			reject(new error.ValidationError('Payload is undefined'));
+			return;
+		}
+
+		const validate = ajv.compile(schema);
+		const valid    = validate(payload);
 
 		if (valid && !validate.errors) {
 			resolve(payload);
@@ -35,11 +40,4 @@ function apiValidator (schema, payload/*, description*/) {
 	});
 }
 
-apiValidator.loadSchemas = parser
-	.dereference(path.resolve('schema/index.json'))
-	.then((schema) => {
-		ajv.addSchema(schema);
-		return schema;
-	});
-
 module.exports = apiValidator;

backend/lib/validator/index.js

@@ -1,17 +1,17 @@
 const _                 = require('lodash');
+const Ajv               = require('ajv/dist/2020');
 const error             = require('../error');
-const definitions = require('../../schema/definitions.json');
+const commonDefinitions = require('../../schema/common.json');
 
 RegExp.prototype.toJSON = RegExp.prototype.toString;
 
-const ajv = require('ajv')({
+const ajv = new Ajv({
-	verbose:     true, //process.env.NODE_ENV === 'development',
+	verbose:         true,
 	allErrors:       true,
-	format:      'full',  // strict regexes for format checks
+	allowUnionTypes: true,
 	coerceTypes:     true,
-	schemas:     [
+	strict:          false,
-		definitions
+	schemas:         [commonDefinitions]
-	]
 });
 
 /**
@@ -27,23 +27,19 @@ function validator (schema, payload) {
 		} else {
 			try {
 				let validate = ajv.compile(schema);
-
 				let valid    = validate(payload);
+
 				if (valid && !validate.errors) {
 					resolve(_.cloneDeep(payload));
 				} else {
 					let message = ajv.errorsText(validate.errors);
 					reject(new error.InternalValidationError(message));
 				}
-
 			} catch (err) {
 				reject(err);
 			}
-
 		}
-
 	});
-
 }
 
 module.exports = validator;

backend/logger.js

@@ -7,6 +7,7 @@ module.exports = {
 	access:    new Signale({scope: 'Access   '}),
 	nginx:     new Signale({scope: 'Nginx    '}),
 	ssl:       new Signale({scope: 'SSL      '}),
+	certbot:   new Signale({scope: 'Certbot  '}),
 	import:    new Signale({scope: 'Importer '}),
 	setup:     new Signale({scope: 'Setup    '}),
 	ip_ranges: new Signale({scope: 'IP Ranges'})

backend/migrations/20190227065017_settings.js

@@ -22,22 +22,6 @@ exports.up = function (knex/*, Promise*/) {
 	})
 		.then(() => {
 			logger.info('[' + migrate_name + '] setting Table created');
-
-			// TODO: add settings
-			let settingModel = require('../models/setting');
-
-			return settingModel
-				.query()
-				.insert({
-					id:          'default-site',
-					name:        'Default Site',
-					description: 'What to show when Nginx is hit with an unknown Host',
-					value:       'congratulations',
-					meta:        {}
-				});
-		})
-		.then(() => {
-			logger.info('[' + migrate_name + '] Default settings added');
 		});
 };
 

backend/migrations/20200410143839_access_list_client.js

@@ -0,0 +1,53 @@
+const migrate_name = 'access_list_client';
+const logger       = require('../logger').migrate;
+
+/**
+ * Migrate
+ *
+ * @see http://knexjs.org/#Schema
+ *
+ * @param   {Object}  knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.up = function (knex/*, Promise*/) {
+
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return knex.schema.createTable('access_list_client', (table) => {
+		table.increments().primary();
+		table.dateTime('created_on').notNull();
+		table.dateTime('modified_on').notNull();
+		table.integer('access_list_id').notNull().unsigned();
+		table.string('address').notNull();
+		table.string('directive').notNull();
+		table.json('meta').notNull();
+
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] access_list_client Table created');
+
+			return knex.schema.table('access_list', function (access_list) {
+				access_list.integer('satify_any').notNull().defaultTo(0);
+			});
+		})
+		.then(() => {
+			logger.info('[' + migrate_name + '] access_list Table altered');
+		});
+};
+
+/**
+ * Undo Migrate
+ *
+ * @param {Object} knex
+ * @param {Promise} Promise
+ * @returns {Promise}
+ */
+exports.down = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Down...');
+
+	return knex.schema.dropTable('access_list_client')
+		.then(() => {
+			logger.info('[' + migrate_name + '] access_list_client Table dropped');
+		});
+};

backend/migrations/20200410143840_access_list_client_fix.js

@@ -0,0 +1,34 @@
+const migrate_name = 'access_list_client_fix';
+const logger       = require('../logger').migrate;
+
+/**
+ * Migrate
+ *
+ * @see http://knexjs.org/#Schema
+ *
+ * @param   {Object}  knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.up = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return knex.schema.table('access_list', function (access_list) {
+		access_list.renameColumn('satify_any', 'satisfy_any');
+	})
+		.then(() => {
+			logger.info('[' + migrate_name + '] access_list Table altered');
+		});
+};
+
+/**
+ * Undo Migrate
+ *
+ * @param   {Object}  knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.down = function (knex, Promise) {
+	logger.warn('[' + migrate_name + '] You can\'t migrate down this one.');
+	return Promise.resolve(true);
+};

backend/migrations/20201014143841_pass_auth.js

@@ -0,0 +1,41 @@
+const migrate_name = 'pass_auth';
+const logger       = require('../logger').migrate;
+
+/**
+ * Migrate
+ *
+ * @see http://knexjs.org/#Schema
+ *
+ * @param   {Object}  knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.up = function (knex/*, Promise*/) {
+
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return knex.schema.table('access_list', function (access_list) {
+		access_list.integer('pass_auth').notNull().defaultTo(1);
+	})
+		.then(() => {
+			logger.info('[' + migrate_name + '] access_list Table altered');
+		});
+};
+
+/**
+ * Undo Migrate
+ *
+ * @param {Object} knex
+ * @param {Promise} Promise
+ * @returns {Promise}
+ */
+exports.down = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Down...');
+
+	return knex.schema.table('access_list', function (access_list) {
+		access_list.dropColumn('pass_auth');
+	})
+		.then(() => {
+			logger.info('[' + migrate_name + '] access_list pass_auth Column dropped');
+		});
+};

backend/migrations/20210210154702_redirection_scheme.js

@@ -0,0 +1,41 @@
+const migrate_name = 'redirection_scheme';
+const logger       = require('../logger').migrate;
+
+/**
+ * Migrate
+ *
+ * @see http://knexjs.org/#Schema
+ *
+ * @param   {Object} knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.up = function (knex/*, Promise*/) {
+
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return knex.schema.table('redirection_host', (table) => {
+		table.string('forward_scheme').notNull().defaultTo('$scheme');
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] redirection_host Table altered');
+		});
+};
+
+/**
+ * Undo Migrate
+ *
+ * @param   {Object} knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.down = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Down...');
+
+	return knex.schema.table('redirection_host', (table) => {
+		table.dropColumn('forward_scheme');
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] redirection_host Table altered');
+		});
+};

backend/migrations/20210210154703_redirection_status_code.js

@@ -0,0 +1,41 @@
+const migrate_name = 'redirection_status_code';
+const logger       = require('../logger').migrate;
+
+/**
+ * Migrate
+ *
+ * @see http://knexjs.org/#Schema
+ *
+ * @param   {Object} knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.up = function (knex/*, Promise*/) {
+
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return knex.schema.table('redirection_host', (table) => {
+		table.integer('forward_http_code').notNull().unsigned().defaultTo(302);
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] redirection_host Table altered');
+		});
+};
+
+/**
+ * Undo Migrate
+ *
+ * @param   {Object} knex
+ * @param   {Promise} Promise
+ * @returns {Promise}
+ */
+exports.down = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Down...');
+
+	return knex.schema.table('redirection_host', (table) => {
+		table.dropColumn('forward_http_code');
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] redirection_host Table altered');
+		});
+};

backend/migrations/20210423103500_stream_domain.js

@@ -0,0 +1,40 @@
+const migrate_name = 'stream_domain';
+const logger       = require('../logger').migrate;
+
+/**
+	* Migrate
+	*
+	* @see http://knexjs.org/#Schema
+	*
+	* @param   {Object} knex
+	* @param   {Promise} Promise
+	* @returns {Promise}
+	*/
+exports.up = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return knex.schema.table('stream', (table) => {
+		table.renameColumn('forward_ip', 'forwarding_host');
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] stream Table altered');
+		});
+};
+
+/**
+	* Undo Migrate
+	*
+	* @param   {Object} knex
+	* @param   {Promise} Promise
+	* @returns {Promise}
+	*/
+exports.down = function (knex/*, Promise*/) {
+	logger.info('[' + migrate_name + '] Migrating Down...');
+
+	return knex.schema.table('stream', (table) => {
+		table.renameColumn('forwarding_host', 'forward_ip');
+	})
+		.then(function () {
+			logger.info('[' + migrate_name + '] stream Table altered');
+		});
+};

backend/migrations/20211108145214_regenerate_default_host.js

@@ -0,0 +1,50 @@
+const migrate_name  = 'stream_domain';
+const logger        = require('../logger').migrate;
+const internalNginx = require('../internal/nginx');
+
+async function regenerateDefaultHost(knex) {
+	const row = await knex('setting').select('*').where('id', 'default-site').first();
+
+	if (!row) {
+		return Promise.resolve();
+	}
+
+	return internalNginx.deleteConfig('default')
+		.then(() => {
+			return internalNginx.generateConfig('default', row);
+		})
+		.then(() => {
+			return internalNginx.test();
+		})
+		.then(() => {
+			return internalNginx.reload();
+		});
+}
+
+/**
+	* Migrate
+	*
+	* @see http://knexjs.org/#Schema
+	*
+	* @param   {Object} knex
+	* @param   {Promise} Promise
+	* @returns {Promise}
+	*/
+exports.up = function (knex) {
+	logger.info('[' + migrate_name + '] Migrating Up...');
+
+	return regenerateDefaultHost(knex);
+};
+
+/**
+	* Undo Migrate
+	*
+	* @param   {Object} knex
+	* @param   {Promise} Promise
+	* @returns {Promise}
+	*/
+exports.down = function (knex) {
+	logger.info('[' + migrate_name + '] Migrating Down...');
+
+	return regenerateDefaultHost(knex);
+};

backend/models/access_list.js

@@ -2,16 +2,25 @@
 // http://vincit.github.io/objection.js/
 
 const db               = require('../db');
+const helpers          = require('../lib/helpers');
 const Model            = require('objection').Model;
 const User             = require('./user');
 const AccessListAuth   = require('./access_list_auth');
+const AccessListClient = require('./access_list_client');
+const now              = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+	'satisfy_any',
+	'pass_auth',
+];
+
 class AccessList extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for meta
 		if (typeof this.meta === 'undefined') {
@@ -20,7 +29,17 @@ class AccessList extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
+	}
+
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
 	}
 
 	static get name () {
@@ -48,7 +67,6 @@ class AccessList extends Model {
 				},
 				modify: function (qb) {
 					qb.where('user.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted', 'email', 'roles']);
 				}
 			},
 			items: {
@@ -57,9 +75,14 @@ class AccessList extends Model {
 				join:       {
 					from: 'access_list.id',
 					to:   'access_list_auth.access_list_id'
+				}
 			},
-				modify: function (qb) {
+			clients: {
-					qb.omit(['id', 'created_on', 'modified_on', 'access_list_id', 'meta']);
+				relation:   Model.HasManyRelation,
+				modelClass: AccessListClient,
+				join:       {
+					from: 'access_list.id',
+					to:   'access_list_client.access_list_id'
 				}
 			},
 			proxy_hosts: {
@@ -71,7 +94,6 @@ class AccessList extends Model {
 				},
 				modify: function (qb) {
 					qb.where('proxy_host.is_deleted', 0);
-					qb.omit(['is_deleted', 'meta']);
 				}
 			}
 		};

backend/models/access_list_auth.js

@@ -3,13 +3,14 @@
 
 const db    = require('../db');
 const Model = require('objection').Model;
+const now   = require('./now_helper');
 
 Model.knex(db);
 
 class AccessListAuth extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for meta
 		if (typeof this.meta === 'undefined') {
@@ -18,7 +19,7 @@ class AccessListAuth extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 	}
 
 	static get name () {
@@ -44,7 +45,6 @@ class AccessListAuth extends Model {
 				},
 				modify: function (qb) {
 					qb.where('access_list.is_deleted', 0);
-					qb.omit(['created_on', 'modified_on', 'is_deleted', 'access_list_id']);
 				}
 			}
 		};

backend/models/access_list_client.js

@@ -0,0 +1,54 @@
+// Objection Docs:
+// http://vincit.github.io/objection.js/
+
+const db    = require('../db');
+const Model = require('objection').Model;
+const now   = require('./now_helper');
+
+Model.knex(db);
+
+class AccessListClient extends Model {
+	$beforeInsert () {
+		this.created_on  = now();
+		this.modified_on = now();
+
+		// Default for meta
+		if (typeof this.meta === 'undefined') {
+			this.meta = {};
+		}
+	}
+
+	$beforeUpdate () {
+		this.modified_on = now();
+	}
+
+	static get name () {
+		return 'AccessListClient';
+	}
+
+	static get tableName () {
+		return 'access_list_client';
+	}
+
+	static get jsonAttributes () {
+		return ['meta'];
+	}
+
+	static get relationMappings () {
+		return {
+			access_list: {
+				relation:   Model.HasOneRelation,
+				modelClass: require('./access_list'),
+				join:       {
+					from: 'access_list_client.access_list_id',
+					to:   'access_list.id'
+				},
+				modify: function (qb) {
+					qb.where('access_list.is_deleted', 0);
+				}
+			}
+		};
+	}
+}
+
+module.exports = AccessListClient;

backend/models/audit-log.js

@@ -4,13 +4,14 @@
 const db    = require('../db');
 const Model = require('objection').Model;
 const User  = require('./user');
+const now   = require('./now_helper');
 
 Model.knex(db);
 
 class AuditLog extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for meta
 		if (typeof this.meta === 'undefined') {
@@ -19,7 +20,7 @@ class AuditLog extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 	}
 
 	static get name () {
@@ -42,9 +43,6 @@ class AuditLog extends Model {
 				join:       {
 					from: 'audit_log.user_id',
 					to:   'user.id'
-				},
-				modify: function (qb) {
-					qb.omit(['id', 'created_on', 'modified_on', 'roles']);
 				}
 			}
 		};

backend/models/auth.js

@@ -3,11 +3,17 @@
 
 const bcrypt  = require('bcrypt');
 const db      = require('../db');
+const helpers = require('../lib/helpers');
 const Model   = require('objection').Model;
 const User    = require('./user');
+const now     = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+];
+
 function encryptPassword () {
 	/* jshint -W040 */
 	let _this = this;
@@ -24,8 +30,8 @@ function encryptPassword () {
 
 class Auth extends Model {
 	$beforeInsert (queryContext) {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for meta
 		if (typeof this.meta === 'undefined') {
@@ -36,10 +42,20 @@ class Auth extends Model {
 	}
 
 	$beforeUpdate (queryContext) {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 		return encryptPassword.apply(this, queryContext);
 	}
 
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
+	}
+
 	/**
 	 * Verify a plain password against the encrypted password
 	 *
@@ -73,9 +89,6 @@ class Auth extends Model {
 				},
 				filter: {
 					is_deleted: 0
-				},
-				modify: function (qb) {
-					qb.omit(['is_deleted']);
 				}
 			}
 		};

backend/models/certificate.js

@@ -2,19 +2,25 @@
 // http://vincit.github.io/objection.js/
 
 const db      = require('../db');
+const helpers = require('../lib/helpers');
 const Model   = require('objection').Model;
 const User    = require('./user');
+const now     = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+];
+
 class Certificate extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for expires_on
 		if (typeof this.expires_on === 'undefined') {
-			this.expires_on = Model.raw('NOW()');
+			this.expires_on = now();
 		}
 
 		// Default for domain_names
@@ -31,7 +37,7 @@ class Certificate extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Sort domain_names
 		if (typeof this.domain_names !== 'undefined') {
@@ -39,6 +45,16 @@ class Certificate extends Model {
 		}
 	}
 
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
+	}
+
 	static get name () {
 		return 'Certificate';
 	}
@@ -62,7 +78,6 @@ class Certificate extends Model {
 				},
 				modify: function (qb) {
 					qb.where('user.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted', 'email', 'roles']);
 				}
 			}
 		};

backend/models/dead_host.js

@@ -2,16 +2,23 @@
 // http://vincit.github.io/objection.js/
 
 const db          = require('../db');
+const helpers     = require('../lib/helpers');
 const Model       = require('objection').Model;
 const User        = require('./user');
 const Certificate = require('./certificate');
+const now         = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+	'enabled',
+];
+
 class DeadHost extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for domain_names
 		if (typeof this.domain_names === 'undefined') {
@@ -27,7 +34,7 @@ class DeadHost extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Sort domain_names
 		if (typeof this.domain_names !== 'undefined') {
@@ -35,6 +42,16 @@ class DeadHost extends Model {
 		}
 	}
 
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
+	}
+
 	static get name () {
 		return 'DeadHost';
 	}
@@ -58,7 +75,6 @@ class DeadHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('user.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted', 'email', 'roles']);
 				}
 			},
 			certificate: {
@@ -70,7 +86,6 @@ class DeadHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('certificate.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted']);
 				}
 			}
 		};

backend/models/now_helper.js

@@ -0,0 +1,13 @@
+const db     = require('../db');
+const config = require('../lib/config');
+const Model  = require('objection').Model;
+
+Model.knex(db);
+
+module.exports = function () {
+	if (config.isSqlite()) {
+		// eslint-disable-next-line
+		return Model.raw("datetime('now','localtime')");
+	}
+	return Model.raw('NOW()');
+};

backend/models/proxy_host.js

@@ -2,17 +2,31 @@
 // http://vincit.github.io/objection.js/
 
 const db          = require('../db');
+const helpers     = require('../lib/helpers');
 const Model       = require('objection').Model;
 const User        = require('./user');
 const AccessList  = require('./access_list');
 const Certificate = require('./certificate');
+const now         = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+	'ssl_forced',
+	'caching_enabled',
+	'block_exploits',
+	'allow_websocket_upgrade',
+	'http2_support',
+	'enabled',
+	'hsts_enabled',
+	'hsts_subdomains',
+];
+
 class ProxyHost extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for domain_names
 		if (typeof this.domain_names === 'undefined') {
@@ -28,7 +42,7 @@ class ProxyHost extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Sort domain_names
 		if (typeof this.domain_names !== 'undefined') {
@@ -36,6 +50,16 @@ class ProxyHost extends Model {
 		}
 	}
 
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
+	}
+
 	static get name () {
 		return 'ProxyHost';
 	}
@@ -59,7 +83,6 @@ class ProxyHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('user.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted', 'email', 'roles']);
 				}
 			},
 			access_list: {
@@ -71,7 +94,6 @@ class ProxyHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('access_list.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted']);
 				}
 			},
 			certificate: {
@@ -83,7 +105,6 @@ class ProxyHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('certificate.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted']);
 				}
 			}
 		};

backend/models/redirection_host.js

@@ -1,17 +1,28 @@
+
 // Objection Docs:
 // http://vincit.github.io/objection.js/
 
 const db          = require('../db');
+const helpers     = require('../lib/helpers');
 const Model       = require('objection').Model;
 const User        = require('./user');
 const Certificate = require('./certificate');
+const now         = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+	'enabled',
+	'preserve_path',
+	'ssl_forced',
+	'block_exploits',
+];
+
 class RedirectionHost extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for domain_names
 		if (typeof this.domain_names === 'undefined') {
@@ -27,7 +38,7 @@ class RedirectionHost extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Sort domain_names
 		if (typeof this.domain_names !== 'undefined') {
@@ -35,6 +46,16 @@ class RedirectionHost extends Model {
 		}
 	}
 
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
+	}
+
 	static get name () {
 		return 'RedirectionHost';
 	}
@@ -58,7 +79,6 @@ class RedirectionHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('user.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted', 'email', 'roles']);
 				}
 			},
 			certificate: {
@@ -70,7 +90,6 @@ class RedirectionHost extends Model {
 				},
 				modify: function (qb) {
 					qb.where('certificate.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted']);
 				}
 			}
 		};

backend/models/stream.js

@@ -2,15 +2,23 @@
 // http://vincit.github.io/objection.js/
 
 const db      = require('../db');
+const helpers = require('../lib/helpers');
 const Model   = require('objection').Model;
 const User    = require('./user');
+const now     = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+	'tcp_forwarding',
+	'udp_forwarding',
+];
+
 class Stream extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for meta
 		if (typeof this.meta === 'undefined') {
@@ -19,7 +27,17 @@ class Stream extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
+	}
+
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
 	}
 
 	static get name () {
@@ -45,7 +63,6 @@ class Stream extends Model {
 				},
 				modify: function (qb) {
 					qb.where('user.is_deleted', 0);
-					qb.omit(['id', 'created_on', 'modified_on', 'is_deleted', 'email', 'roles']);
 				}
 			}
 		};

backend/models/token.js

@@ -4,36 +4,38 @@
  */
 
 const _      = require('lodash');
-const config = require('config');
 const jwt    = require('jsonwebtoken');
 const crypto = require('crypto');
+const config = require('../lib/config');
 const error  = require('../lib/error');
+const logger = require('../logger').global;
 const ALGO   = 'RS256';
 
 module.exports = function () {
-	const public_key  = config.get('jwt.pub');
-	const private_key = config.get('jwt.key');
 
 	let token_data = {};
 
-	let self = {
+	const self = {
 		/**
 		 * @param {Object}  payload
 		 * @returns {Promise}
 		 */
 		create: (payload) => {
+			if (!config.getPrivateKey()) {
+				logger.error('Private key is empty!');
+			}
 			// sign with RSA SHA256
-			let options = {
+			const options = {
 				algorithm: ALGO,
 				expiresIn: payload.expiresIn || '1d'
 			};
 
 			payload.jti = crypto.randomBytes(12)
 				.toString('base64')
-				.substr(-8);
+				.substring(-8);
 
 			return new Promise((resolve, reject) => {
-				jwt.sign(payload, private_key, options, (err, token) => {
+				jwt.sign(payload, config.getPrivateKey(), options, (err, token) => {
 					if (err) {
 						reject(err);
 					} else {
@@ -52,12 +54,15 @@ module.exports = function () {
 		 * @returns {Promise}
 		 */
 		load: function (token) {
+			if (!config.getPublicKey()) {
+				logger.error('Public key is empty!');
+			}
 			return new Promise((resolve, reject) => {
 				try {
 					if (!token || token === null || token === 'null') {
 						reject(new error.AuthError('Empty token'));
 					} else {
-						jwt.verify(token, public_key, {ignoreExpiration: false, algorithms: [ALGO]}, (err, result) => {
+						jwt.verify(token, config.getPublicKey(), {ignoreExpiration: false, algorithms: [ALGO]}, (err, result) => {
 							if (err) {
 
 								if (err.name === 'TokenExpiredError') {
@@ -72,8 +77,6 @@ module.exports = function () {
 								// Hack: some tokens out in the wild have a scope of 'all' instead of 'user'.
 								// For 30 days at least, we need to replace 'all' with user.
 								if ((typeof token_data.scope !== 'undefined' && _.indexOf(token_data.scope, 'all') !== -1)) {
-									//console.log('Warning! Replacing "all" scope with "user"');
-
 									token_data.scope = ['user'];
 								}
 
@@ -123,7 +126,7 @@ module.exports = function () {
 		 * @returns {Integer}
 		 */
 		getUserId: (default_value) => {
-			let attrs = self.get('attrs');
+			const attrs = self.get('attrs');
 			if (attrs && typeof attrs.id !== 'undefined' && attrs.id) {
 				return attrs.id;
 			}

backend/models/user.js

@@ -2,15 +2,22 @@
 // http://vincit.github.io/objection.js/
 
 const db             = require('../db');
+const helpers        = require('../lib/helpers');
 const Model          = require('objection').Model;
 const UserPermission = require('./user_permission');
+const now            = require('./now_helper');
 
 Model.knex(db);
 
+const boolFields = [
+	'is_deleted',
+	'is_disabled',
+];
+
 class User extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 
 		// Default for roles
 		if (typeof this.roles === 'undefined') {
@@ -19,7 +26,17 @@ class User extends Model {
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
+	}
+
+	$parseDatabaseJson(json) {
+		json = super.$parseDatabaseJson(json);
+		return helpers.convertIntFieldsToBool(json, boolFields);
+	}
+
+	$formatDatabaseJson(json) {
+		json = helpers.convertBoolFieldsToInt(json, boolFields);
+		return super.$formatDatabaseJson(json);
 	}
 
 	static get name () {
@@ -42,9 +59,6 @@ class User extends Model {
 				join:       {
 					from: 'user.id',
 					to:   'user_permission.user_id'
-				},
-				modify: function (qb) {
-					qb.omit(['id', 'created_on', 'modified_on', 'user_id']);
 				}
 			}
 		};

backend/models/user_permission.js

@@ -3,17 +3,18 @@
 
 const db    = require('../db');
 const Model = require('objection').Model;
+const now   = require('./now_helper');
 
 Model.knex(db);
 
 class UserPermission extends Model {
 	$beforeInsert () {
-		this.created_on  = Model.raw('NOW()');
+		this.created_on  = now();
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 	}
 
 	$beforeUpdate () {
-		this.modified_on = Model.raw('NOW()');
+		this.modified_on = now();
 	}
 
 	static get name () {

backend/package.json

@@ -2,34 +2,31 @@
 	"name": "nginx-proxy-manager",
 	"version": "0.0.0",
 	"description": "A beautiful interface for creating Nginx endpoints",
-	"main": "js/index.js",
+	"main": "index.js",
 	"dependencies": {
-		"ajv": "^6.11.0",
+		"@apidevtools/json-schema-ref-parser": "^11.7.0",
+		"ajv": "^8.17.1",
+		"archiver": "^5.3.0",
 		"batchflow": "^0.4.0",
-		"bcrypt": "^3.0.8",
+		"bcrypt": "^5.0.0",
-		"body-parser": "^1.19.0",
+		"body-parser": "^1.20.3",
 		"compression": "^1.7.4",
-		"config": "^3.2.5",
+		"express": "^4.20.0",
-		"diskdb": "^0.1.17",
+		"express-fileupload": "^1.1.9",
-		"express": "^4.17.1",
-		"express-fileupload": "^1.1.6",
 		"gravatar": "^1.8.0",
-		"html-entities": "^1.2.1",
+		"jsonwebtoken": "^9.0.0",
-		"json-schema-ref-parser": "^7.1.3",
+		"knex": "2.4.2",
-		"jsonwebtoken": "^8.5.1",
+		"liquidjs": "10.6.1",
-		"knex": "^0.20.10",
+		"lodash": "^4.17.21",
-		"liquidjs": "^9.7.1",
+		"moment": "^2.29.4",
-		"lodash": "^4.17.15",
+		"mysql2": "^3.11.1",
-		"moment": "^2.24.0",
+		"node-rsa": "^1.0.8",
-		"mysql": "^2.18.1",
+		"objection": "3.0.1",
-		"node-rsa": "^1.0.7",
-		"nodemon": "^2.0.2",
-		"objection": "^2.1.3",
 		"path": "^0.12.7",
-		"restler": "^3.4.0",
+		"pg": "^8.13.0",
-		"signale": "^1.4.0",
+		"signale": "1.4.0",
-		"temp-write": "^4.0.0",
+		"sqlite3": "5.1.6",
-		"unix-timestamp": "^0.2.0"
+		"temp-write": "^4.0.0"
 	},
 	"signale": {
 		"displayDate": true,
@@ -38,8 +35,14 @@
 	"author": "Jamie Curnow <jc@jc21.com>",
 	"license": "MIT",
 	"devDependencies": {
-		"eslint": "^6.8.0",
+		"@apidevtools/swagger-parser": "^10.1.0",
+		"chalk": "4.1.2",
+		"eslint": "^8.36.0",
 		"eslint-plugin-align-assignments": "^1.1.2",
-		"prettier": "^1.19.1"
+		"nodemon": "^2.0.2",
+		"prettier": "^2.0.4"
+	},
+	"scripts": {
+		"validate-schema": "node validate-schema.js"
 	}
 }

backend/routes/audit-log.js

@@ -1,7 +1,7 @@
 const express          = require('express');
-const validator        = require('../../lib/validator');
+const validator        = require('../lib/validator');
-const jwtdecode        = require('../../lib/express/jwt-decode');
+const jwtdecode        = require('../lib/express/jwt-decode');
-const internalAuditLog = require('../../internal/audit-log');
+const internalAuditLog = require('../internal/audit-log');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -14,7 +14,7 @@ let router = express.Router({
  */
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -29,10 +29,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {

backend/routes/main.js

@@ -1,6 +1,6 @@
 const express = require('express');
-const pjson   = require('../../package.json');
+const pjson   = require('../package.json');
-const error   = require('../../lib/error');
+const error   = require('../lib/error');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -43,7 +43,7 @@ router.use('/nginx/certificates', require('./nginx/certificates'));
  *
  * ALL /api/*
  */
-router.all(/(.+)/, function (req, res, next) {
+router.all(/(.+)/, function (req, _, next) {
 	req.params.page = req.params['0'];
 	next(new error.ItemNotFoundError(req.params.page));
 });

backend/routes/nginx/access_lists.js

@@ -1,8 +1,9 @@
 const express            = require('express');
-const validator          = require('../../../lib/validator');
+const validator          = require('../../lib/validator');
-const jwtdecode          = require('../../../lib/express/jwt-decode');
+const jwtdecode          = require('../../lib/express/jwt-decode');
-const internalAccessList = require('../../../internal/access-list');
+const apiValidator       = require('../../lib/validator/api');
-const apiValidator       = require('../../../lib/validator/api');
+const internalAccessList = require('../../internal/access-list');
+const schema             = require('../../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -30,10 +31,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -56,7 +57,7 @@ router
 	 * Create a new access-list
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/access-lists#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/access-lists', 'post'), req.body)
 			.then((payload) => {
 				return internalAccessList.create(res.locals.access, payload);
 			})
@@ -74,7 +75,7 @@ router
  */
 router
 	.route('/:list_id')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -90,10 +91,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				list_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -119,7 +120,7 @@ router
 	 * Update and existing access-list
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/access-lists#/links/2/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/access-lists/{listID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = parseInt(req.params.list_id, 10);
 				return internalAccessList.update(res.locals.access, payload);

backend/routes/nginx/certificates.js

@@ -1,8 +1,10 @@
 const express             = require('express');
-const validator           = require('../../../lib/validator');
+const error               = require('../../lib/error');
-const jwtdecode           = require('../../../lib/express/jwt-decode');
+const validator           = require('../../lib/validator');
-const internalCertificate = require('../../../internal/certificate');
+const jwtdecode           = require('../../lib/express/jwt-decode');
-const apiValidator        = require('../../../lib/validator/api');
+const apiValidator        = require('../../lib/validator/api');
+const internalCertificate = require('../../internal/certificate');
+const schema              = require('../../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -15,7 +17,7 @@ let router = express.Router({
  */
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -30,10 +32,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -56,8 +58,9 @@ router
 	 * Create a new certificate
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/certificates#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/certificates', 'post'), req.body)
 			.then((payload) => {
+				req.setTimeout(900000); // 15 minutes timeout
 				return internalCertificate.create(res.locals.access, payload);
 			})
 			.then((result) => {
@@ -67,6 +70,37 @@ router
 			.catch(next);
 	});
 
+/**
+ * Test HTTP challenge for domains
+ *
+ * /api/nginx/certificates/test-http
+ */
+router
+	.route('/test-http')
+	.options((_, res) => {
+		res.sendStatus(204);
+	})
+	.all(jwtdecode())
+
+	/**
+	 * GET /api/nginx/certificates/test-http
+	 *
+	 * Test HTTP challenge for domains
+	 */
+	.get((req, res, next) => {
+		if (req.query.domains === undefined) {
+			next(new error.ValidationError('Domains are required as query parameters'));
+			return;
+		}
+
+		internalCertificate.testHttpsChallenge(res.locals.access, JSON.parse(req.query.domains))
+			.then((result) => {
+				res.status(200)
+					.send(result);
+			})
+			.catch(next);
+	});
+
 /**
  * Specific certificate
  *
@@ -74,7 +108,7 @@ router
  */
 router
 	.route('/:certificate_id')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -90,10 +124,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				certificate_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -113,24 +147,6 @@ router
 			.catch(next);
 	})
 
-	/**
-	 * PUT /api/nginx/certificates/123
-	 *
-	 * Update and existing certificate
-	 */
-	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/certificates#/links/2/schema'}, req.body)
-			.then((payload) => {
-				payload.id = parseInt(req.params.certificate_id, 10);
-				return internalCertificate.update(res.locals.access, payload);
-			})
-			.then((result) => {
-				res.status(200)
-					.send(result);
-			})
-			.catch(next);
-	})
-
 	/**
 	 * DELETE /api/nginx/certificates/123
 	 *
@@ -152,7 +168,7 @@ router
  */
 router
 	.route('/:certificate_id/upload')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -186,7 +202,7 @@ router
  */
 router
 	.route('/:certificate_id/renew')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -197,6 +213,7 @@ router
 	 * Renew certificate
 	 */
 	.post((req, res, next) => {
+		req.setTimeout(900000); // 15 minutes timeout
 		internalCertificate.renew(res.locals.access, {
 			id: parseInt(req.params.certificate_id, 10)
 		})
@@ -207,6 +224,34 @@ router
 			.catch(next);
 	});
 
+/**
+ * Download LE Certs
+ *
+ * /api/nginx/certificates/123/download
+ */
+router
+	.route('/:certificate_id/download')
+	.options((req, res) => {
+		res.sendStatus(204);
+	})
+	.all(jwtdecode())
+
+	/**
+	 * GET /api/nginx/certificates/123/download
+	 *
+	 * Renew certificate
+	 */
+	.get((req, res, next) => {
+		internalCertificate.download(res.locals.access, {
+			id: parseInt(req.params.certificate_id, 10)
+		})
+			.then((result) => {
+				res.status(200)
+					.download(result.fileName);
+			})
+			.catch(next);
+	});
+
 /**
  * Validate Certs before saving
  *
@@ -214,7 +259,7 @@ router
  */
 router
 	.route('/validate')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())

backend/routes/nginx/dead_hosts.js

@@ -1,8 +1,9 @@
 const express          = require('express');
-const validator        = require('../../../lib/validator');
+const validator        = require('../../lib/validator');
-const jwtdecode        = require('../../../lib/express/jwt-decode');
+const jwtdecode        = require('../../lib/express/jwt-decode');
-const internalDeadHost = require('../../../internal/dead-host');
+const apiValidator     = require('../../lib/validator/api');
-const apiValidator     = require('../../../lib/validator/api');
+const internalDeadHost = require('../../internal/dead-host');
+const schema           = require('../../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -15,7 +16,7 @@ let router = express.Router({
  */
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -30,10 +31,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -56,7 +57,7 @@ router
 	 * Create a new dead-host
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/dead-hosts#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/dead-hosts', 'post'), req.body)
 			.then((payload) => {
 				return internalDeadHost.create(res.locals.access, payload);
 			})
@@ -90,10 +91,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				host_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -119,7 +120,7 @@ router
 	 * Update and existing dead-host
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/dead-hosts#/links/2/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/dead-hosts/{hostID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = parseInt(req.params.host_id, 10);
 				return internalDeadHost.update(res.locals.access, payload);
@@ -152,7 +153,7 @@ router
  */
 router
 	.route('/:host_id/enable')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -176,7 +177,7 @@ router
  */
 router
 	.route('/:host_id/disable')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())

backend/routes/nginx/proxy_hosts.js

@@ -1,8 +1,9 @@
 const express           = require('express');
-const validator         = require('../../../lib/validator');
+const validator         = require('../../lib/validator');
-const jwtdecode         = require('../../../lib/express/jwt-decode');
+const jwtdecode         = require('../../lib/express/jwt-decode');
-const internalProxyHost = require('../../../internal/proxy-host');
+const apiValidator      = require('../../lib/validator/api');
-const apiValidator      = require('../../../lib/validator/api');
+const internalProxyHost = require('../../internal/proxy-host');
+const schema            = require('../../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -30,10 +31,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -56,7 +57,7 @@ router
 	 * Create a new proxy-host
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/proxy-hosts#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/proxy-hosts', 'post'), req.body)
 			.then((payload) => {
 				return internalProxyHost.create(res.locals.access, payload);
 			})
@@ -90,10 +91,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				host_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -119,7 +120,7 @@ router
 	 * Update and existing proxy-host
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/proxy-hosts#/links/2/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/proxy-hosts/{hostID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = parseInt(req.params.host_id, 10);
 				return internalProxyHost.update(res.locals.access, payload);
@@ -152,7 +153,7 @@ router
  */
 router
 	.route('/:host_id/enable')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -176,7 +177,7 @@ router
  */
 router
 	.route('/:host_id/disable')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())

backend/routes/nginx/redirection_hosts.js

@@ -1,8 +1,9 @@
 const express                 = require('express');
-const validator               = require('../../../lib/validator');
+const validator               = require('../../lib/validator');
-const jwtdecode               = require('../../../lib/express/jwt-decode');
+const jwtdecode               = require('../../lib/express/jwt-decode');
-const internalRedirectionHost = require('../../../internal/redirection-host');
+const apiValidator            = require('../../lib/validator/api');
-const apiValidator            = require('../../../lib/validator/api');
+const internalRedirectionHost = require('../../internal/redirection-host');
+const schema                  = require('../../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -30,10 +31,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -56,7 +57,7 @@ router
 	 * Create a new redirection-host
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/redirection-hosts#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/redirection-hosts', 'post'), req.body)
 			.then((payload) => {
 				return internalRedirectionHost.create(res.locals.access, payload);
 			})
@@ -90,10 +91,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				host_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -119,7 +120,7 @@ router
 	 * Update and existing redirection-host
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/redirection-hosts#/links/2/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/redirection-hosts/{hostID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = parseInt(req.params.host_id, 10);
 				return internalRedirectionHost.update(res.locals.access, payload);

backend/routes/nginx/streams.js

@@ -1,8 +1,9 @@
 const express        = require('express');
-const validator      = require('../../../lib/validator');
+const validator      = require('../../lib/validator');
-const jwtdecode      = require('../../../lib/express/jwt-decode');
+const jwtdecode      = require('../../lib/express/jwt-decode');
-const internalStream = require('../../../internal/stream');
+const apiValidator   = require('../../lib/validator/api');
-const apiValidator   = require('../../../lib/validator/api');
+const internalStream = require('../../internal/stream');
+const schema         = require('../../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -30,10 +31,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -56,7 +57,7 @@ router
 	 * Create a new stream
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/streams#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/streams', 'post'), req.body)
 			.then((payload) => {
 				return internalStream.create(res.locals.access, payload);
 			})
@@ -90,10 +91,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				stream_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -119,7 +120,7 @@ router
 	 * Update and existing stream
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/streams#/links/2/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/nginx/streams/{streamID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = parseInt(req.params.stream_id, 10);
 				return internalStream.update(res.locals.access, payload);
@@ -152,7 +153,7 @@ router
  */
 router
 	.route('/:host_id/enable')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -176,7 +177,7 @@ router
  */
 router
 	.route('/:host_id/disable')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())

backend/routes/reports.js

@@ -1,6 +1,6 @@
 const express        = require('express');
-const jwtdecode      = require('../../lib/express/jwt-decode');
+const jwtdecode      = require('../lib/express/jwt-decode');
-const internalReport = require('../../internal/report');
+const internalReport = require('../internal/report');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -10,14 +10,14 @@ let router = express.Router({
 
 router
 	.route('/hosts')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 
 	/**
 	 * GET /reports/hosts
 	 */
-	.get(jwtdecode(), (req, res, next) => {
+	.get(jwtdecode(), (_, res, next) => {
 		internalReport.getHostsReport(res.locals.access)
 			.then((data) => {
 				res.status(200)

backend/routes/schema.js

@@ -1,8 +1,8 @@
 const express = require('express');
-const swaggerJSON = require('../../doc/api.swagger.json');
+const schema  = require('../schema');
-const PACKAGE     = require('../../package.json');
+const PACKAGE = require('../package.json');
 
-let router = express.Router({
+const router = express.Router({
 	caseSensitive: true,
 	strict:        true,
 	mergeParams:   true
@@ -10,14 +10,16 @@ let router = express.Router({
 
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 
 	/**
 	 * GET /schema
 	 */
-	.get((req, res/*, next*/) => {
+	.get(async (req, res) => {
+		let swaggerJSON = await schema.getCompiledSchema();
+
 		let proto = req.protocol;
 		if (typeof req.headers['x-forwarded-proto'] !== 'undefined' && req.headers['x-forwarded-proto']) {
 			proto = req.headers['x-forwarded-proto'];

backend/routes/settings.js

@@ -1,8 +1,9 @@
 const express         = require('express');
-const validator       = require('../../lib/validator');
+const validator       = require('../lib/validator');
-const jwtdecode       = require('../../lib/express/jwt-decode');
+const jwtdecode       = require('../lib/express/jwt-decode');
-const internalSetting = require('../../internal/setting');
+const apiValidator    = require('../lib/validator/api');
-const apiValidator    = require('../../lib/validator/api');
+const internalSetting = require('../internal/setting');
+const schema          = require('../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -15,7 +16,7 @@ let router = express.Router({
  */
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -25,7 +26,7 @@ router
 	 *
 	 * Retrieve all settings
 	 */
-	.get((req, res, next) => {
+	.get((_, res, next) => {
 		internalSetting.getAll(res.locals.access)
 			.then((rows) => {
 				res.status(200)
@@ -41,7 +42,7 @@ router
  */
 router
 	.route('/:setting_id')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -57,7 +58,8 @@ router
 			additionalProperties: false,
 			properties:           {
 				setting_id: {
-					$ref: 'definitions#/definitions/setting_id'
+					type:      'string',
+					minLength: 1
 				}
 			}
 		}, {
@@ -81,7 +83,7 @@ router
 	 * Update and existing setting
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/settings#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/settings/{settingID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = req.params.setting_id;
 				return internalSetting.update(res.locals.access, payload);

backend/routes/tokens.js

@@ -1,7 +1,8 @@
 const express       = require('express');
-const jwtdecode     = require('../../lib/express/jwt-decode');
+const jwtdecode     = require('../lib/express/jwt-decode');
-const internalToken = require('../../internal/token');
+const apiValidator  = require('../lib/validator/api');
-const apiValidator  = require('../../lib/validator/api');
+const internalToken = require('../internal/token');
+const schema        = require('../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -11,7 +12,7 @@ let router = express.Router({
 
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 
@@ -39,11 +40,9 @@ router
 	 *
 	 * Create a new Token
 	 */
-	.post((req, res, next) => {
+	.post(async (req, res, next) => {
-		apiValidator({$ref: 'endpoints/tokens#/links/0/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/tokens', 'post'), req.body)
-			.then((payload) => {
+			.then(internalToken.getTokenFromEmail)
-				return internalToken.getTokenFromEmail(payload);
-			})
 			.then((data) => {
 				res.status(200)
 					.send(data);

backend/routes/users.js

@@ -1,9 +1,10 @@
 const express      = require('express');
-const validator    = require('../../lib/validator');
+const validator    = require('../lib/validator');
-const jwtdecode    = require('../../lib/express/jwt-decode');
+const jwtdecode    = require('../lib/express/jwt-decode');
-const userIdFromMe = require('../../lib/express/user-id-from-me');
+const userIdFromMe = require('../lib/express/user-id-from-me');
-const internalUser = require('../../internal/user');
+const internalUser = require('../internal/user');
-const apiValidator = require('../../lib/validator/api');
+const apiValidator = require('../lib/validator/api');
+const schema       = require('../schema');
 
 let router = express.Router({
 	caseSensitive: true,
@@ -16,7 +17,7 @@ let router = express.Router({
  */
 router
 	.route('/')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -31,10 +32,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				},
 				query: {
-					$ref: 'definitions#/definitions/query'
+					$ref: 'common#/properties/query'
 				}
 			}
 		}, {
@@ -48,7 +49,11 @@ router
 				res.status(200)
 					.send(users);
 			})
-			.catch(next);
+			.catch((err) => {
+				console.log(err);
+				next(err);
+			});
+		//.catch(next);
 	})
 
 	/**
@@ -57,7 +62,7 @@ router
 	 * Create a new User
 	 */
 	.post((req, res, next) => {
-		apiValidator({$ref: 'endpoints/users#/links/1/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/users', 'post'), req.body)
 			.then((payload) => {
 				return internalUser.create(res.locals.access, payload);
 			})
@@ -75,7 +80,7 @@ router
  */
 router
 	.route('/:user_id')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())
@@ -92,10 +97,10 @@ router
 			additionalProperties: false,
 			properties:           {
 				user_id: {
-					$ref: 'definitions#/definitions/id'
+					$ref: 'common#/properties/id'
 				},
 				expand: {
-					$ref: 'definitions#/definitions/expand'
+					$ref: 'common#/properties/expand'
 				}
 			}
 		}, {
@@ -113,7 +118,10 @@ router
 				res.status(200)
 					.send(user);
 			})
-			.catch(next);
+			.catch((err) => {
+				console.log(err);
+				next(err);
+			});
 	})
 
 	/**
@@ -122,7 +130,7 @@ router
 	 * Update and existing user
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/users#/links/2/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/users/{userID}', 'put'), req.body)
 			.then((payload) => {
 				payload.id = req.params.user_id;
 				return internalUser.update(res.locals.access, payload);
@@ -167,7 +175,7 @@ router
 	 * Update password for a user
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/users#/links/4/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/users/{userID}/auth', 'put'), req.body)
 			.then((payload) => {
 				payload.id = req.params.user_id;
 				return internalUser.setPassword(res.locals.access, payload);
@@ -198,7 +206,7 @@ router
 	 * Set some or all permissions for a user
 	 */
 	.put((req, res, next) => {
-		apiValidator({$ref: 'endpoints/users#/links/5/schema'}, req.body)
+		apiValidator(schema.getValidationSchema('/users/{userID}/permissions', 'put'), req.body)
 			.then((payload) => {
 				payload.id = req.params.user_id;
 				return internalUser.setPermissions(res.locals.access, payload);
@@ -217,7 +225,7 @@ router
  */
 router
 	.route('/:user_id/login')
-	.options((req, res) => {
+	.options((_, res) => {
 		res.sendStatus(204);
 	})
 	.all(jwtdecode())

backend/schema/common.json

@@ -0,0 +1,115 @@
+{
+	"$schema": "https://json-schema.org/draft/2020-12/schema",
+	"$id": "common",
+	"type": "object",
+	"properties": {
+		"id": {
+			"description": "Unique identifier",
+			"readOnly": true,
+			"type": "integer",
+			"minimum": 1
+		},
+		"expand": {
+			"anyOf": [
+				{
+					"type": "null"
+				},
+				{
+					"type": "array",
+					"minItems": 1,
+					"items": {
+						"type": "string"
+					}
+				}
+			]
+		},
+		"query": {
+			"anyOf": [
+				{
+					"type": "null"
+				},
+				{
+					"type": "string",
+					"minLength": 1,
+					"maxLength": 255
+				}
+			]
+		},
+		"created_on": {
+			"description": "Date and time of creation",
+			"readOnly": true,
+			"type": "string"
+		},
+		"modified_on": {
+			"description": "Date and time of last update",
+			"readOnly": true,
+			"type": "string"
+		},
+		"user_id": {
+			"description": "User ID",
+			"type": "integer",
+			"minimum": 1
+		},
+		"certificate_id": {
+			"description": "Certificate ID",
+			"anyOf": [
+				{
+					"type": "integer",
+					"minimum": 0
+				},
+				{
+					"type": "string",
+					"pattern": "^new$"
+				}
+			]
+		},
+		"access_list_id": {
+			"description": "Access List ID",
+			"type": "integer",
+			"minimum": 0
+		},
+		"domain_names": {
+			"description": "Domain Names separated by a comma",
+			"type": "array",
+			"minItems": 1,
+			"maxItems": 100,
+			"uniqueItems": true,
+			"items": {
+				"type": "string",
+				"pattern": "^[^&| @!#%^();:/\\\\}{=+?<>,~`'\"]+$"
+			}
+		},
+		"enabled": {
+			"description": "Is Enabled",
+			"type": "boolean"
+		},
+		"ssl_forced": {
+			"description": "Is SSL Forced",
+			"type": "boolean"
+		},
+		"hsts_enabled": {
+			"description": "Is HSTS Enabled",
+			"type": "boolean"
+		},
+		"hsts_subdomains": {
+			"description": "Is HSTS applicable to all subdomains",
+			"type": "boolean"
+		},
+		"ssl_provider": {
+			"type": "string",
+			"pattern": "^(letsencrypt|other)$"
+		},
+		"http2_support": {
+			"description": "HTTP2 Protocol Support",
+			"type": "boolean"
+		},
+		"block_exploits": {
+			"description": "Should we block common exploits",
+			"type": "boolean"
+		},
+		"caching_enabled": {
+			"description": "Should we cache assets",
+			"type": "boolean"
+		}
+	}
+}

backend/schema/components/access-list-object.json

@@ -0,0 +1,53 @@
+{
+	"type": "object",
+	"description": "Access List object",
+	"required": ["id", "created_on", "modified_on", "owner_user_id", "name", "directive", "address", "satisfy_any", "pass_auth", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"owner_user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"name": {
+			"type": "string",
+			"minLength": 1
+		},
+		"directive": {
+			"type": "string",
+			"enum": ["allow", "deny"]
+		},
+		"address": {
+			"oneOf": [
+				{
+					"type": "string",
+					"pattern": "^([0-9]{1,3}\\.){3}[0-9]{1,3}(/([0-9]|[1-2][0-9]|3[0-2]))?$"
+				},
+				{
+					"type": "string",
+					"pattern": "^s*((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]d|1dd|[1-9]?d)(.(25[0-5]|2[0-4]d|1dd|[1-9]?d)){3}))|:)))(%.+)?s*(/([0-9]|[1-9][0-9]|1[0-1][0-9]|12[0-8]))?$"
+				},
+				{
+					"type": "string",
+					"pattern": "^all$"
+				}
+			]
+		},
+		"satisfy_any": {
+			"type": "boolean"
+		},
+		"pass_auth": {
+			"type": "boolean"
+		},
+		"meta": {
+			"type": "object"
+		}
+	}
+}

backend/schema/components/audit-log-object.json

@@ -0,0 +1,32 @@
+{
+	"type": "object",
+	"description": "Audit Log object",
+	"required": ["id", "created_on", "modified_on", "user_id", "object_type", "object_id", "action", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"object_type": {
+			"type": "string"
+		},
+		"object_id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"action": {
+			"type": "string"
+		},
+		"meta": {
+			"type": "object"
+		}
+	}
+}

backend/schema/components/certificate-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "Certificates list",
+	"items": {
+		"$ref": "./certificate-object.json"
+	}
+}

backend/schema/components/certificate-object.json

@@ -0,0 +1,64 @@
+{
+	"type": "object",
+	"description": "Certificate object",
+	"required": ["id", "created_on", "modified_on", "owner_user_id", "provider", "nice_name", "domain_names", "expires_on", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"owner_user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"provider": {
+			"$ref": "../common.json#/properties/ssl_provider"
+		},
+		"nice_name": {
+			"type": "string",
+			"description": "Nice Name for the custom certificate"
+		},
+		"domain_names": {
+			"$ref": "../common.json#/properties/domain_names"
+		},
+		"expires_on": {
+			"description": "Date and time of expiration",
+			"readOnly": true,
+			"type": "string"
+		},
+		"meta": {
+			"type": "object",
+			"additionalProperties": false,
+			"properties": {
+				"letsencrypt_email": {
+					"type": "string"
+				},
+				"letsencrypt_agree": {
+					"type": "boolean"
+				},
+				"dns_challenge": {
+					"type": "boolean"
+				},
+				"dns_provider": {
+					"type": "string"
+				},
+				"dns_provider_credentials": {
+					"type": "string"
+				},
+				"propagation_seconds": {
+					"anyOf": [
+						{
+							"type": "integer",
+							"minimum": 0
+						}
+					]
+				}
+			}
+		}
+	}
+}

backend/schema/components/dead-host-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "404 Hosts list",
+	"items": {
+		"$ref": "./dead-host-object.json"
+	}
+}

backend/schema/components/dead-host-object.json

@@ -0,0 +1,47 @@
+{
+	"type": "object",
+	"description": "404 Host object",
+	"required": ["id", "created_on", "modified_on", "owner_user_id", "domain_names", "certificate_id", "ssl_forced", "hsts_enabled", "hsts_subdomains", "http2_support", "advanced_config", "enabled", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"owner_user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"domain_names": {
+			"$ref": "../common.json#/properties/domain_names"
+		},
+		"certificate_id": {
+			"$ref": "../common.json#/properties/certificate_id"
+		},
+		"ssl_forced": {
+			"$ref": "../common.json#/properties/ssl_forced"
+		},
+		"hsts_enabled": {
+			"$ref": "../common.json#/properties/hsts_enabled"
+		},
+		"hsts_subdomains": {
+			"$ref": "../common.json#/properties/hsts_subdomains"
+		},
+		"http2_support": {
+			"$ref": "../common.json#/properties/http2_support"
+		},
+		"advanced_config": {
+			"type": "string"
+		},
+		"enabled": {
+			"$ref": "../common.json#/properties/enabled"
+		},
+		"meta": {
+			"type": "object"
+		}
+	}
+}

backend/schema/components/error-object.json

@@ -0,0 +1,14 @@
+{
+	"type": "object",
+	"description": "Error object",
+	"additionalProperties": false,
+	"required": ["code", "message"],
+	"properties": {
+		"code": {
+			"type": "integer"
+		},
+		"message": {
+			"type": "string"
+		}
+	}
+}

backend/schema/components/error.json

@@ -0,0 +1,9 @@
+{
+	"type": "object",
+	"description": "Error",
+	"properties": {
+		"error": {
+			"$ref": "./error-object.json"
+		}
+	}
+}

backend/schema/components/health-object.json

@@ -0,0 +1,38 @@
+{
+	"type": "object",
+	"description": "Health object",
+	"additionalProperties": false,
+	"required": ["status", "version"],
+	"properties": {
+		"status": {
+			"type": "string",
+			"description": "Healthy",
+			"example": "OK"
+		},
+		"version": {
+			"type": "object",
+			"description": "The version object",
+			"example": {
+				"major": 2,
+				"minor": 0,
+				"revision": 0
+			},
+			"additionalProperties": false,
+			"required": ["major", "minor", "revision"],
+			"properties": {
+				"major": {
+					"type": "integer",
+					"minimum": 0
+				},
+				"minor": {
+					"type": "integer",
+					"minimum": 0
+				},
+				"revision": {
+					"type": "integer",
+					"minimum": 0
+				}
+			}
+		}
+	}
+}

backend/schema/components/permission-object.json

@@ -0,0 +1,41 @@
+{
+	"type": "object",
+	"minProperties": 1,
+	"properties": {
+		"visibility": {
+			"type": "string",
+			"description": "Visibility Type",
+			"enum": ["all", "user"]
+		},
+		"access_lists": {
+			"type": "string",
+			"description": "Access Lists Permissions",
+			"enum": ["hidden", "view", "manage"]
+		},
+		"dead_hosts": {
+			"type": "string",
+			"description": "404 Hosts Permissions",
+			"enum": ["hidden", "view", "manage"]
+		},
+		"proxy_hosts": {
+			"type": "string",
+			"description": "Proxy Hosts Permissions",
+			"enum": ["hidden", "view", "manage"]
+		},
+		"redirection_hosts": {
+			"type": "string",
+			"description": "Redirection Permissions",
+			"enum": ["hidden", "view", "manage"]
+		},
+		"streams": {
+			"type": "string",
+			"description": "Streams Permissions",
+			"enum": ["hidden", "view", "manage"]
+		},
+		"certificates": {
+			"type": "string",
+			"description": "Certificates Permissions",
+			"enum": ["hidden", "view", "manage"]
+		}
+	}
+}

backend/schema/components/proxy-host-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "Proxy Hosts list",
+	"items": {
+		"$ref": "./proxy-host-object.json"
+	}
+}

backend/schema/components/proxy-host-object.json

@@ -0,0 +1,162 @@
+{
+	"type": "object",
+	"description": "Proxy Host object",
+	"required": [
+		"id",
+		"created_on",
+		"modified_on",
+		"owner_user_id",
+		"domain_names",
+		"forward_host",
+		"forward_port",
+		"access_list_id",
+		"certificate_id",
+		"ssl_forced",
+		"caching_enabled",
+		"block_exploits",
+		"advanced_config",
+		"meta",
+		"allow_websocket_upgrade",
+		"http2_support",
+		"forward_scheme",
+		"enabled",
+		"locations",
+		"hsts_enabled",
+		"hsts_subdomains",
+		"certificate",
+		"use_default_location",
+		"ipv6"
+	],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"owner_user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"domain_names": {
+			"$ref": "../common.json#/properties/domain_names"
+		},
+		"forward_host": {
+			"type": "string",
+			"minLength": 1,
+			"maxLength": 255
+		},
+		"forward_port": {
+			"type": "integer",
+			"minimum": 1,
+			"maximum": 65535
+		},
+		"access_list_id": {
+			"$ref": "../common.json#/properties/access_list_id"
+		},
+		"certificate_id": {
+			"$ref": "../common.json#/properties/certificate_id"
+		},
+		"ssl_forced": {
+			"$ref": "../common.json#/properties/ssl_forced"
+		},
+		"caching_enabled": {
+			"$ref": "../common.json#/properties/caching_enabled"
+		},
+		"block_exploits": {
+			"$ref": "../common.json#/properties/block_exploits"
+		},
+		"advanced_config": {
+			"type": "string"
+		},
+		"meta": {
+			"type": "object"
+		},
+		"allow_websocket_upgrade": {
+			"description": "Allow Websocket Upgrade for all paths",
+			"example": true,
+			"type": "boolean"
+		},
+		"http2_support": {
+			"$ref": "../common.json#/properties/http2_support"
+		},
+		"forward_scheme": {
+			"type": "string",
+			"enum": ["http", "https"]
+		},
+		"enabled": {
+			"$ref": "../common.json#/properties/enabled"
+		},
+		"locations": {
+			"type": "array",
+			"minItems": 0,
+			"items": {
+				"type": "object",
+				"required": ["forward_scheme", "forward_host", "forward_port", "path"],
+				"additionalProperties": false,
+				"properties": {
+					"id": {
+						"type": ["integer", "null"]
+					},
+					"path": {
+						"type": "string",
+						"minLength": 1
+					},
+					"forward_scheme": {
+						"$ref": "#/properties/forward_scheme"
+					},
+					"forward_host": {
+						"$ref": "#/properties/forward_host"
+					},
+					"forward_port": {
+						"$ref": "#/properties/forward_port"
+					},
+					"forward_path": {
+						"type": "string"
+					},
+					"advanced_config": {
+						"type": "string"
+					}
+				}
+			}
+		},
+		"hsts_enabled": {
+			"$ref": "../common.json#/properties/hsts_enabled"
+		},
+		"hsts_subdomains": {
+			"$ref": "../common.json#/properties/hsts_subdomains"
+		},
+		"certificate": {
+			"oneOf": [
+				{
+					"type": "null"
+				},
+				{
+					"$ref": "./certificate-object.json"
+				}
+			]
+		},
+		"owner": {
+			"$ref": "./user-object.json"
+		},
+		"access_list": {
+			"oneOf": [
+				{
+					"type": "null"
+				},
+				{
+					"$ref": "./access-list-object.json"
+				}
+			]
+		},
+		"use_default_location": {
+			"type": "boolean"
+		},
+		"ipv6": {
+			"type": "boolean"
+		}
+	}
+}

backend/schema/components/redirection-host-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "Redirection Hosts list",
+	"items": {
+		"$ref": "./redirection-host-object.json"
+	}
+}

backend/schema/components/redirection-host-object.json

@@ -0,0 +1,72 @@
+{
+	"type": "object",
+	"description": "Redirection Host object",
+	"required": ["id", "created_on", "modified_on", "owner_user_id", "domain_names", "forward_http_code", "forward_scheme", "forward_domain_name", "preserve_path", "certificate_id", "ssl_forced", "hsts_enabled", "hsts_subdomains", "http2_support", "block_exploits", "advanced_config", "enabled", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"owner_user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"domain_names": {
+			"$ref": "../common.json#/properties/domain_names"
+		},
+		"forward_http_code": {
+			"description": "Redirect HTTP Status Code",
+			"example": 302,
+			"type": "integer",
+			"minimum": 300,
+			"maximum": 308
+		},
+		"forward_scheme": {
+			"type": "string",
+			"enum": ["http", "https"]
+		},
+		"forward_domain_name": {
+			"description": "Domain Name",
+			"example": "jc21.com",
+			"type": "string",
+			"pattern": "^(?:[^.*]+\\.?)+[^.]$"
+		},
+		"preserve_path": {
+			"description": "Should the path be preserved",
+			"example": true,
+			"type": "boolean"
+		},
+		"certificate_id": {
+			"$ref": "../common.json#/properties/certificate_id"
+		},
+		"ssl_forced": {
+			"$ref": "../common.json#/properties/ssl_forced"
+		},
+		"hsts_enabled": {
+			"$ref": "../common.json#/properties/hsts_enabled"
+		},
+		"hsts_subdomains": {
+			"$ref": "../common.json#/properties/hsts_subdomains"
+		},
+		"http2_support": {
+			"$ref": "../common.json#/properties/http2_support"
+		},
+		"block_exploits": {
+			"$ref": "../common.json#/properties/block_exploits"
+		},
+		"advanced_config": {
+			"type": "string"
+		},
+		"enabled": {
+			"$ref": "../common.json#/properties/enabled"
+		},
+		"meta": {
+			"type": "object"
+		}
+	}
+}

backend/schema/components/security-schemes.json

@@ -0,0 +1,6 @@
+{
+	"BearerAuth": {
+		"type": "http",
+		"scheme": "bearer"
+	}
+}

backend/schema/components/setting-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "Setting list",
+	"items": {
+		"$ref": "./setting-object.json"
+	}
+}

backend/schema/components/setting-object.json

@@ -0,0 +1,53 @@
+{
+	"type": "object",
+	"description": "Setting object",
+	"required": ["id", "name", "description", "value", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"type": "string",
+			"description": "Setting ID",
+			"minLength": 1,
+			"example": "default-site"
+		},
+		"name": {
+			"type": "string",
+			"description": "Setting Display Name",
+			"minLength": 1,
+			"example": "Default Site"
+		},
+		"description": {
+			"type": "string",
+			"description": "Meaningful description",
+			"minLength": 1,
+			"example": "What to show when Nginx is hit with an unknown Host"
+		},
+		"value": {
+			"description": "Value in almost any form",
+			"example": "congratulations",
+			"oneOf": [
+				{
+					"type": "string",
+					"minLength": 1
+				},
+				{
+					"type": "integer"
+				},
+				{
+					"type": "object"
+				},
+				{
+					"type": "number"
+				},
+				{
+					"type": "array"
+				}
+			]
+		},
+		"meta": {
+			"description": "Extra metadata",
+			"example": {},
+			"type": "object"
+		}
+	}
+}

backend/schema/components/stream-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "Proxy Hosts list",
+	"items": {
+		"$ref": "./proxy-host-object.json"
+	}
+}

backend/schema/components/stream-object.json

@@ -0,0 +1,60 @@
+{
+	"type": "object",
+	"description": "Stream object",
+	"required": ["id", "created_on", "modified_on", "owner_user_id", "incoming_port", "forwarding_host", "forwarding_port", "tcp_forwarding", "udp_forwarding", "enabled", "meta"],
+	"additionalProperties": false,
+	"properties": {
+		"id": {
+			"$ref": "../common.json#/properties/id"
+		},
+		"created_on": {
+			"$ref": "../common.json#/properties/created_on"
+		},
+		"modified_on": {
+			"$ref": "../common.json#/properties/modified_on"
+		},
+		"owner_user_id": {
+			"$ref": "../common.json#/properties/user_id"
+		},
+		"incoming_port": {
+			"type": "integer",
+			"minimum": 1,
+			"maximum": 65535
+		},
+		"forwarding_host": {
+			"anyOf": [
+				{
+					"description": "Domain Name",
+					"example": "jc21.com",
+					"type": "string",
+					"pattern": "^(?:[^.*]+\\.?)+[^.]$"
+				},
+				{
+					"type": "string",
+					"format": "ipv4"
+				},
+				{
+					"type": "string",
+					"format": "ipv6"
+				}
+			]
+		},
+		"forwarding_port": {
+			"type": "integer",
+			"minimum": 1,
+			"maximum": 65535
+		},
+		"tcp_forwarding": {
+			"type": "boolean"
+		},
+		"udp_forwarding": {
+			"type": "boolean"
+		},
+		"enabled": {
+			"$ref": "../common.json#/properties/enabled"
+		},
+		"meta": {
+			"type": "object"
+		}
+	}
+}

backend/schema/components/token-object.json

@@ -0,0 +1,19 @@
+{
+	"type": "object",
+	"description": "Token object",
+	"required": ["expires", "token"],
+	"additionalProperties": false,
+	"properties": {
+		"expires": {
+			"description": "Token Expiry Unix Time",
+			"example": 1566540249,
+			"minimum": 1,
+			"type": "number"
+		},
+		"token": {
+			"description": "JWT Token",
+			"example": "eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.ey...xaHKYr3Kk6MvkUjcC4",
+			"type": "string"
+		}
+	}
+}

backend/schema/components/user-list.json

@@ -0,0 +1,7 @@
+{
+	"type": "array",
+	"description": "User list",
+	"items": {
+		"$ref": "./user-object.json"
+	}
+}