Space not starting up after production configuration
Hi all,
I'm very new to space on premises and have been trying to install it locally on Ubuntu 22.04 LTS. Please bear with me, because I don't have much experience in this and really need some help, because I'm a bit stumped.
Running docker compose works fine without any issues and I can get to the UI at http://127.0.0.1:8084.
Then configuring space for production and extending it to be accessible outside the network, I used this documentation:
https://www.jetbrains.com/help/space/configure-space-for-docker-compose-production-environment.html
Essentially there is only a few steps to follow concerning the update of the config files and creating a config file for Nginx. I can reach the space site externally after the config updates, but I'm getting a 502 error. All certs are correct and the connection is secure, but Nginx is returning a bad gateway.
I checked the Nginx logs at /var/log/nginx/error.log and I get this error message:
2023/08/15 08:45:15 [error] 340202#340202: *197 connect() failed (111: Unknown error) while connecting to upstream, client: {ommitted}, server: space.securotrax.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8084/", host: "space.securotrax.com"
Investigating further, I found that http://127.0.0.1:8084 no longer works. And I don't know why. I also don't know where to find the space logs to try and see where the problem might be and I'm stuck at this stage.
Here is my Nginx config file (/etc/nginx/conf.d/space.conf):
server {
client_max_body_size 0;
server_name space.securotrax.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:8084/;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
listen 443 ssl;
ssl_certificate /opt/space/tls/certs/space-cert.crt;
ssl_certificate_key /opt/space/tls/keys/space.key;
}
server {
client_max_body_size 0;
server_name git.securotrax.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:8080/;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
listen 443 ssl;
ssl_certificate /opt/space/tls/certs/git-cert.crt;
ssl_certificate_key /opt/space/tls/keys/git.key;
}
server {
client_max_body_size 0;
server_name packages.securotrax.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:8390/;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
listen 443 ssl;
ssl_certificate /opt/space/tls/certs/packages-cert.crt;
ssl_certificate_key /opt/space/tls/keys/packages.key;
}
server {
client_max_body_size 0;
server_name minio.securotrax.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://127.0.0.1:9000/;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
listen 443 ssl;
ssl_certificate /opt/space/tls/certs/packages-cert.crt;
ssl_certificate_key /opt/space/tls/keys/packages.key;
}
# configuring HTTP redirects
server {
if ($host = space.securotrax.com) {
return 301 https://$host$request_uri;
}
server_name space.securotrax.com;
listen 80;
return 404;
}
server {
if ($host = git.securotrax.com) {
return 301 https://$host$request_uri;
}
server_name git.securotrax.com;
listen 80;
return 404;
}
server {
if ($host = packages.securotrax.com) {
return 301 https://$host$request_uri;
}
server_name packages.securotrax.com;
listen 80;
return 404;
}
server {
if ($host = minio.securotrax.com) {
return 301 https://$host$request_uri;
}
server_name minio.securotrax.com;
listen 80;
return 404;
}
And my space.on-premises.conf file:
ktor {
deployment {
environment = production
port = 8084
internalPort = 9084
autoreload = false
callGroupSize = 8
}
application {
modules = [circlet.server.ServerKt.main]
}
}
circlet {
stackName = "On prem"
type = "Docker"
server {
singleOrg = true
features = [OnPremise]
enabledFeatureFlags = [PROJECT_PARAMETERS_ANNOTATIONS, PROJECT_PARAMETERS_VAULT_INTEGRATION, GOTO_MENTIONS_NEW_MATCH_ALGORITHM, GOTO_SEARCH_PROFILES_FTS, FULL_TEXT_SEARCH_EXPERIMENTAL_UI, ADVANCED_SEARCH_EXPERIMENTAL_UI, ISSUE_CHAT_CONTENT, POSTPONED_CHANNEL_MESSAGES]
enabledFeatureFlags += ${?EXTRA_FEATURE_FLAG}
enableSqlTracing = false
enableExternalApiTracing = false
enableAutomationInternalsConfiguration = true
enableActiveAppTracker = true
dumpStacksOnExit = false
httpApiTimeout = 60000
dontFailOnLargeBatches = true
rpcTimeout = 60000
permissionCacheCalculationInterval = 2000
dbCacheCalculationTimeoutMs = 60000
pidFileName = ".main_app.pid"
orgInitTimeout = 180000
processMetricsEnabled = false
}
onprem {
jetSalesURL = "https://account.jetbrains.com"
}
push {
proxy {
url = "https://on-premise.service.jetbrains.space"
}
}
preload {
defaultProviders = [PreloadSpaceFeedAdvertiser, PreloadGeneralChannel, PreloadAbsenceReasons, PreloadMainProject]
preloadMainProject = true
builtinAuthModule {
enabled = true
}
admin {
firstName = "admin"
lastName = "admin"
username = omitted
email = omitted
password = omitted
}
}
masterSecret = omitted
db {
type = "postgresql"
schemaPrefix = "crclt_"
host = "postgres"
port = 5432
port = ${?DB_PORT}
username = "space"
password = omitted
postgresql {
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://"${circlet.db.host}":"${circlet.db.port}"/spacedb"
}
pool {
vendor = "hikari"
maxActive = 96
minIdle = 12
}
debug {
similar_queries_count_threshold = 10
all_queries_count_threshold = 100
}
}
bus {
engine = "redis"
host = "redis://redis:6379"
type = "single"
}
storage {
aws {
region = "eu-west-1"
bucket = "space-local"
endpoint = "http://minio:9000"
accessKey = omitted
secretKey = omitted
}
elastic {
enabled = true
host = "elasticsearch"
port = 9200
reIndex = true
}
}
audit {
elastic {
enabled = true
host = "elasticsearch"
port = 9200
}
}
perf {
client {
enabled = false
}
elastic {
enabled = false
}
riemann {
enabled = false
}
prometheus {
enabled = false
jmx = false
}
}
metrics {
fus {
enabled = true
config = "https://resources.jetbrains.com/storage/fus/config/v4/SP/SPP.json"
product = "SPP"
}
}
// Circlet as OAuth 2.0 authentication server
oauth {
rememberMe {
// Remember me cookie TTL in seconds
ttl {
// If user checked remember me
userLong = 2592000 // 30 * 24 * 60 * 60
// If admin checked remember me
adminLong = 604800 // 7 * 24 * 60 * 60
// If user or admin unchecked remember me
short = 3600 // 60 * 60
}
// Authentication session cache
cache {
protectedQueueSize = 100
probationalQueueSize = 100
}
}
accessToken {
// Access token TTL in seconds
ttl = 600 // do not use anything longer in production, potential security issue
// List of client IDs with custom TTL for access_token
ttlCustom {
"circlet-ios-app" = 25h
"circlet-android-app" = 25h
}
issuer = "https://jetbrains.com/circlet"
// RSA512 key pair to sign JWT
// Use circlet.platform.server.services.oauth.accessToken.AccessTokenKt.main to generate the pair
rsa {
// X.509 encoded public key
public =omitted
// PKCS#8 encoded private key
private =omitted
}
}
message {
encoding {
// an example of 128 bit key for AES.
// openssl rand 16 | base64
key = omitted
}
}
twoFactor {
// an example of 128 bit key for AES.
// openssl rand 16 | base64
encryptionKey = omitted
}
// 128 bit key for AES.
encryptionKey = omitted
// 2048-bit public/private key pair for signing/verifying outgoing messages to applications
// Use circlet.test.framework.application.KeyGenUtilsKt.main to generate the pair
messageSigning {
rsa {
// X.509 encoded public key
public =omitted
// PKCS#8 encoded private key
private =omitted
}
}
}
limits {
perOrg = {
"totalMembers" = {
softCap = 9000000
hardCap = 10000000
}
}
request {
sessionTTL = 5000
limits = [
{
type = "bandwidth"
capacity = 1000
refill {
tokens = 400
duration = "1 second"
}
},
{
type = "bandwidth"
capacity = 10000
refill {
tokens = 10000
duration = "1 minute"
}
}
]
}
}
# license {
# key = {{ .Values.space.license.key }}
# }
vcs {
url = "http://vcs:8080"
secret = omitted
identity {
name = "Space"
email = "space@jetbrains.com"
secretKey_base64 = omitted
passphrase = ""
}
}
mail {
outgoing {
enabled = true
// SMTP settings
fromAddress = omitted
host = omitted
port = 25
protocol = "SMTP"
login = omitted
password = omitted
messageQueuePrefix = "mailQueue"
// handling properties
aggregationDelaySecs = 900
rateLimitPerSecond = 3
}
}
endUserAgreement {
jetBrains {
enabled = false
}
}
frontend {
url = "https://space.securotrax.com"
internalUrl = "http://space:9084"
internalALB {
enabled = false
}
// Alternative URLs for Space
altUrls = [
"https://localhost:8084",
]
enableWebdriverTestIds = false
}
healthcheck {
username = "healthcheck"
password = "healthcheck"
}
webhooks {
key = omitted
// Possible options are "Java" and "Apache", unknown values fall back to Java.
client = "Java"
}
unfurl {
// Possible options are "Java" and "Apache", unknown values fall back to Apache.
client = "Apache"
}
// external language server. This implementation implies syntax highlight only
// `client` should be one of `net` or `local`
language-server {
client = local
engine = Java
host = langservice
port = 8095
}
pipelines {
dsl {
compilation {
image = "public.registry.jetbrains.space/p/space/containers/space-automation-dsl-compile:2023.2.0.128"
storage {
aws {
region = "eu-west-1"
bucket = "automation-dsl-local"
endpoint = "http://minio:9000"
accessKey = omitted
secretKey = omitted
}
}
}
runtime {
mavenRepo {
url = "https://maven.pkg.jetbrains.space/public/p/space/maven"
}
}
}
execution {
type = "docker"
backends = ["worker"]
# TODO remove when space container is updated
bootstrap {
image = "671090787922.dkr.ecr.eu-west-1.amazonaws.com/automation/alpine/git:v2.30.2-jb"
}
worker {
image = "public.registry.jetbrains.space/p/space/containers/space-automation-worker:2023.2.0.128"
# workers won't be updated after installing new on-prem version
initialVersion = "default"
dockerBuilderImage = "public.registry.jetbrains.space/p/space/containers/space-automation-docker-builder:2023.2.0.128"
dslCompileImage = "public.registry.jetbrains.space/p/space/containers/space-automation-dsl-compile:2023.2.0.128"
extraNetworks = ["space-on-premises_backend-data"]
extraHosts = ["host.docker.internal:host-gateway"]
}
terminateLongExecutions {
maxPendingDuration = 6h
maxScheduledDuration = 30m
}
fileshare {
storage {
aws {
region = "eu-west-1"
bucket = "automation-fileshare-local"
endpoint = "http://minio:9000"
accessKey = omitted
secretKey = omitted
s3Presigner {
endpoint = "http://minio:9000"
}
}
}
}
}
serviceMessages {
storage {
aws {
region = "eu-west-1"
bucket = "automation-logs-local"
endpoint = "http://minio:9000"
accessKey = omitted
secretKey = omitted
}
}
}
blockstorage {
engine = "local-docker"
}
}
packages {
notifications {
enabled = true
}
oauth {
clientId = omitted
clientSecret = omitted
}
types {
maven {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
nuget {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
npm {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
container {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
pypi {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
composer {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
dart {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
files {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
crates {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
}
}
}
attachments {
useCdn = false
}
}
And the packages.on-premises.conf file:
ktor {
deployment {
environment = production
port = 8390
internalPort = 9390
autoreload = false
callGroupSize = 8
}
application {
modules = [circlet.packages.server.ServerKt.main]
}
}
circlet {
bus {
engine = "redis"
host = "redis://redis:6379"
type = "single"
}
organization {
name = "JetBrains"
slogan = "The Drive to Develop"
schema = "spacedb"
domain = "jetbrains"
}
server {
configurableLogging = false
processMetricsEnabled = false
}
masterSecret = omitted
db {
type = "postgresql"
schemaPrefix = "crclt_"
host = "postgres"
username = "omitted"
password = "omitted"
postgresql {
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://postgres:5432/spacedb"
}
pool {
vendor = "hikari"
maxActive = 96
minIdle = 12
}
debug {
similar_queries_count_threshold = 10
all_queries_count_threshold = 100
}
}
packages {
url = "https://packages.securotrax.com"
internalUrl = "http://packages:9390"
supportedTypes = "container, maven, nuget, npm, pypi, composer, files, dart, crates"
altUrls = [
"http://localhost:8390",
"http://localhost:9390",
]
oauth {
clientId = omitted
clientSecret = omitted
}
vulnerabilities {
url = ""
}
}
perf {
client {
enabled = false
}
elastic {
enabled = false
}
riemann {
enabled = false
}
prometheus {
enabled = false
jmx = false
}
}
space {
url = "https://space.securotrax.com"
orgMode = "SINGLE_ORG"
internalUrl = "http://space:9084"
}
storage {
aws {
region = "eu-west-1"
bucket = "packages-local"
endpoint = "http://minio:9000"
publicUrl = "https://minio.securotrax.com"
accessKey = "omitted"
secretKey = "omitted"
}
elastic {
enabled = true
host = "elasticsearch"
port = 9200
reIndex = true
}
}
}
And also the vcs.on-premises.properties file
base.url=https://git.securotrax.com
vcs.git.hosted=true
vcs.ssh.git.hosted=true
vcs.ssh.git.port=2222
circlet.enabled=true
circlet.key={omitted}
data-dir=/home/space/git/vcs-hosting/data
disable-access-log=true
id=VCS-hosting
listen=0.0.0.0
port=8080
logs-dir=logs
presentation-service-name=VCS-hosting
temp-dir=temp
circlet.url.int=http://space:9084
circlet.url.ext=https://space.securotrax.com
vcs.dfs.storage=s3
vcs.dfs.s3_data_bucket_name=vcs-local
vcs.dfs.s3_data_region=eu-west-1
vcs.dfs.s3_data_access_key={omitted}
vcs.dfs.s3_data_secret_key={omitted}
vcs.dfs.s3_endpoint=http://minio:9000
vcs.dfs.s3_offload=false
vcs.dfs.cache.path=/home/space/git/vcs-hosting/cache
vcs.dfs.sql_url=jdbc:postgresql://postgres:5432/spacedb
vcs.dfs.sql_schema=vcsdfs
vcs.dfs.sql_username={omitted}
vcs.dfs.sql_password={omitted}
vcs.dfs.redis=single
vcs.dfs.redis_url=redis://redis:6379
vcs.ssh.server.key.rsa.base64={omitted}
vcs.gpg.public_key.base64={omitted}
I think that there might be problem in the configs, but I have no idea what it could be. Any help or suggestions would be greatly appreciated.
Please sign in to leave a comment.
Techadmin, thanks for reaching out to us! You're right, as you're getting a 5** error, it's necessary to check if Space works as expected. For this, let's access the logs:
1. get the IDs of Space-related containers: `docker ps`;
2. access logs for space, packages and vcs containers: `docker logs <container_id>`.
Please share the outputs with me and I'll check the details. In case you don't want to share the logs publicly, feel free to submit a support request using this link: https://jb.gg/space-support. Thanks!
Thanks Pavel Boger
It seems like the container starts, but then crashes:
Here is the result of sudo docker ps:
The Space container is the top one and then I rant the sudo docker logs b7f2611f93ca command as suggested. I then saw the two exceptions listed below. Also noticed now that it completely crashes space because running the docker ps command again shows this where the space container is no longer running:
So this surely must be a mistake in the configuration somewhere. I just don't know where? Could you possibly point in a direction? Thanks so much for your help so far!
Techadmin I believe you unintentionally removed a `circlet.packages.internalUrl = "http://packages:9390"` from the `space.on-premises.conf` file. Could you put it back and try to start Space again?
Thanks so much Pavel Boger
Just to confirm... should it look like this?
Pavel Boger You're right... I did remove it, but now can't remember exactly where to place it back :)
I removed it because the jetbrains documentation did not have it in. Here is a copy from the docs:
Sorry it looks a bit messy, but that's how it copies from the site. The above does not have it in there and as a result, I thought it had to be removed.
Techadmin, right, it should be like this:
circlet {...
packages {
internalUrl = "http://packages:9390"
...
}
}
Thanks Pavel Boger, I've made the change and it's working correctly now! Thanks so much for all your help!