[OnPremKubernetes] How to use Computeservice?

Avatar
Till Rother
Created

Hey, I followed the docs on enabling computeservice and have been able to get it to manage the automation namespace but now I'm stuck at getting them to actually accept jobs from Space.

It currently seems like Computeservice and Space don't really communicate with each other. 

For one, do I set the default worker pool to Space Automation Cloud or to Self-Hosted Workers?

How to I correctly refer to the computeservice in the requirements section of the .kts-file?

 

Votes

0

Share

18 comments
Avatar
Till Rother
Created

Alright, I did some testing and checked the logs of multiple workers, and found that the worker pods can't connect to http://jb-space-computeservice.kube-space:9098.
I spun up a busybox and tried to connect to that URL from there but wasn't able to reach it, not even from the same namespace.

I'm checking my cluster-config and cni next

0
Avatar
Till Rother
Created

Current error message the workers write:

10:08:57.447 INFO  Downloading file from http://jb-space-computeservice.kube-space:9098/resources/worker/impl/latest/zip...
10:08:57.499 INFO  Downloaded 77 / 77 bytes (100%), average speed is 77B/s
10:08:57.511 INFO  Problem during http request, attempt #0
io.ktor.client.plugins.ServerResponseException: Server error(GET http://jb-space-computeservice.kube-space:9098/resources/worker/impl/latest/zip: 500 Server Error. Text: "{"error":"internal-server-error","error_description":"Internal Server Error"}"
        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:52)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at circlet.pipelines.updater.UpdaterMainKt$runUpdater$1.invokeSuspend$lambda$4(UpdaterMain.kt:120)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
0
Avatar
Till Rother
Created

Tried to access the URL from the computeservice-pod using curl:

curl http://localhost:9098/resources/worker/impl/latest/zip

Error message:

{"error":"internal-server-error","error_description":"Internal Server Error"}

I get the same when using the following:

curl http://jb-space-computeservice.kube-space:9098/resources/worker/impl/latest/zip

When I was playing around with automation during Beta, the packages-pod was the one that provided those automation-worker.zip-files, did that change?

 

0
Avatar
Till Rother
Created

This is the error the computeservice-pod writes when it starts a worker:

2023-01-24 12:17:25.849 [ktor-jetty-8098-9098-4] ERROR Application [trace_id=7192191795435380590] - Server error
java.lang.IllegalStateException: 'ComputeService' doesn't contain a component for type circlet.compute.server.resources.ComputeBinariesService [jdk.internal.loader.ClassLoaders$AppClassLoader@7ad041f3]
        at circlet.compute.server.resources.ComputeBinariesServiceKt.getBinaries(ComputeBinariesService.kt:5)
        at circlet.compute.server.resources.ResourceRoutesKt$resourceRoutes$1.invokeSuspend(ResourceRoutes.kt:24)
        at circlet.compute.server.resources.ResourceRoutesKt$resourceRoutes$1.invoke(ResourceRoutes.kt)
        at circlet.compute.server.resources.ResourceRoutesKt$resourceRoutes$1.invoke(ResourceRoutes.kt)
        at io.ktor.server.routing.Route$buildPipeline$1$1.invokeSuspend(Route.kt:116)
        at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
        at io.ktor.server.routing.Route$buildPipeline$1$1.invoke(Route.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at circlet.compute.server.application.ComputeServerApplication$callContextInterceptor$1$invokeSuspend$$inlined$with$1.invokeSuspend(MDC.kt:6)
        at circlet.compute.server.application.ComputeServerApplication$callContextInterceptor$1$invokeSuspend$$inlined$with$1.invoke(MDC.kt)
        at circlet.compute.server.application.ComputeServerApplication$callContextInterceptor$1$invokeSuspend$$inlined$with$1.invoke(MDC.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:169)
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
        at circlet.compute.server.application.ComputeServerApplication$callContextInterceptor$1.invokeSuspend(ComputeServerApplication.kt:32)
        at circlet.compute.server.application.ComputeServerApplication$callContextInterceptor$1.invoke(ComputeServerApplication.kt)
        at circlet.compute.server.application.ComputeServerApplication$callContextInterceptor$1.invoke(ComputeServerApplication.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at circlet.compute.server.routes.RoutesKt$rawHttpApiMetricInterceptor$1.invokeSuspend(Routes.kt:9)
        at circlet.compute.server.routes.RoutesKt$rawHttpApiMetricInterceptor$1.invoke(Routes.kt)
        at circlet.compute.server.routes.RoutesKt$rawHttpApiMetricInterceptor$1.invoke(Routes.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$5.invokeSuspend(ServerKt$platformMain$1$4$6$5.java:39)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$5.a(ServerKt$platformMain$1$4$6$5.java)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$5.invoke(ServerKt$platformMain$1$4$6$5.java)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at circlet.platform.a.q.m.a(m.java:33)
        at circlet.platform.a.q.m.a(m.java:21)
        at circlet.platform.a.q.m$b$a.invokeSuspend(m$b$a.java:2)
        at circlet.platform.a.q.m$b$a.a(m$b$a.java)
        at circlet.platform.a.q.m$b$a.invoke(m$b$a.java)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$b$a.invokeSuspend(ServerKt$platformMain$1$4$6$b$a.java:4)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$b$a.a(ServerKt$platformMain$1$4$6$b$a.java)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$b$a.invoke(ServerKt$platformMain$1$4$6$b$a.java)
        at circlet.platform.a.a.PlatformApplication$b.invokeSuspend(PlatformApplication$b.java:6)
        at circlet.platform.a.a.PlatformApplication$b.a(PlatformApplication$b.java)
        at circlet.platform.a.a.PlatformApplication$b.invoke(PlatformApplication$b.java)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:169)
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
        at circlet.platform.a.a.PlatformApplication.a(PlatformApplication.java:15)
        at circlet.platform.a.a.PlatformApplication.a(PlatformApplication.java)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$b.invokeSuspend(ServerKt$platformMain$1$4$6$b.java:5)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$b.a(ServerKt$platformMain$1$4$6$b.java)
        at circlet.platform.a.ServerKt$platformMain$1$4$6$b.invoke(ServerKt$platformMain$1$4$6$b.java)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
        at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
        at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
        at io.ktor.server.routing.Routing$executeResult$$inlined$execute$1.invoke(Pipeline.kt)
        at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
        at io.ktor.server.routing.Routing.executeResult(Routing.kt:190)
        at io.ktor.server.routing.Routing.interceptor(Routing.kt:64)
        at io.ktor.server.routing.Routing$Plugin$install$1.invokeSuspend(Routing.kt:140)
        at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
        at io.ktor.server.routing.Routing$Plugin$install$1.invoke(Routing.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invokeSuspend(BaseApplicationEngine.kt:123)
        at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
        at io.ktor.server.engine.BaseApplicationEngineKt$installDefaultTransformationChecker$1.invoke(BaseApplicationEngine.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at circlet.platform.a.q.e.a(e.java:20)
        at circlet.platform.a.q.e.a(e.java:25)
        at circlet.platform.a.q.e$b$c.invokeSuspend(e$b$c.java:2)
        at circlet.platform.a.q.e$b$c.a(e$b$c.java)
        at circlet.platform.a.q.e$b$c.invoke(e$b$c.java)
        at circlet.platform.tracing.TraceExecutionKt$traceExecutionAsRoot$$inlined$executeWithSpanAndFinish$1.invokeSuspend(ExecuteWithSpan.kt:14)
        at circlet.platform.tracing.TraceExecutionKt$traceExecutionAsRoot$$inlined$executeWithSpanAndFinish$1.invoke(ExecuteWithSpan.kt)
        at circlet.platform.tracing.TraceExecutionKt$traceExecutionAsRoot$$inlined$executeWithSpanAndFinish$1.invoke(ExecuteWithSpan.kt)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2$1.invokeSuspend(withTracingContext.kt:25)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2$1.invoke(withTracingContext.kt)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2$1.invoke(withTracingContext.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:169)
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2.invokeSuspend(withTracingContext.kt:24)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2.invoke(withTracingContext.kt)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2.invoke(withTracingContext.kt)
        at circlet.platform.tracing.WithTraceMDCKt$withTraceMDC$$inlined$with$1.invokeSuspend(MDC.kt:70)
        at circlet.platform.tracing.WithTraceMDCKt$withTraceMDC$$inlined$with$1.invoke(MDC.kt)
        at circlet.platform.tracing.WithTraceMDCKt$withTraceMDC$$inlined$with$1.invoke(MDC.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:169)
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
        at circlet.platform.tracing.WithTraceMDCKt.withTraceMDC(withTraceMDC.kt:54)
        at circlet.platform.tracing.WithTracingContextKt.withTracingContext(withTracingContext.kt:23)
        at circlet.platform.tracing.TraceExecutionKt.traceExecutionAsRoot(TraceExecution.kt:52)
        at circlet.platform.a.q.e$b.invokeSuspend(e$b.java:10)
        at circlet.platform.a.q.e$b.a(e$b.java)
        at circlet.platform.a.q.e$b.invoke(e$b.java)
        at circlet.platform.tracing.UtilKt$withTracingTags$3.invokeSuspend(Util.kt:58)
        at circlet.platform.tracing.UtilKt$withTracingTags$3.invoke(Util.kt)
        at circlet.platform.tracing.UtilKt$withTracingTags$3.invoke(Util.kt)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2$1.invokeSuspend(withTracingContext.kt:25)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2$1.invoke(withTracingContext.kt)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2$1.invoke(withTracingContext.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:169)
        at kotlinx.coroutines.BuildersKt.withContext(Unknown Source)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2.invokeSuspend(withTracingContext.kt:24)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2.invoke(withTracingContext.kt)
        at circlet.platform.tracing.WithTracingContextKt$withTracingContext$2.invoke(withTracingContext.kt)
        at circlet.platform.tracing.WithTraceMDCKt.withTraceMDC(withTraceMDC.kt:7)
        at circlet.platform.tracing.WithTracingContextKt.withTracingContext(withTracingContext.kt:23)
        at circlet.platform.tracing.UtilKt.withTracingTags(Util.kt:57)
        at circlet.platform.tracing.UtilKt.withTracingTags(Util.kt:42)
        at circlet.platform.a.q.e.a(e.java:3)
        at circlet.platform.a.q.d$b.invokeSuspend(d$b.java:3)
        at circlet.platform.a.q.d$b.a(d$b.java)
        at circlet.platform.a.q.d$b.invoke(d$b.java)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at io.ktor.server.application.hooks.CallFailed$install$1$1.invokeSuspend(CommonHooks.kt:43)
        at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
        at io.ktor.server.application.hooks.CallFailed$install$1$1.invoke(CommonHooks.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89)
        at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:264)
        at io.ktor.server.application.hooks.CallFailed$install$1.invokeSuspend(CommonHooks.kt:42)
        at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
        at io.ktor.server.application.hooks.CallFailed$install$1.invoke(CommonHooks.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
        at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invokeSuspend(DefaultEnginePipeline.kt:118)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline$1.invoke(DefaultEnginePipeline.kt)
        at io.ktor.util.pipeline.SuspendFunctionGun.loop(SuspendFunctionGun.kt:120)
        at io.ktor.util.pipeline.SuspendFunctionGun.proceed(SuspendFunctionGun.kt:78)
        at io.ktor.util.pipeline.SuspendFunctionGun.execute$ktor_utils(SuspendFunctionGun.kt:98)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:77)
        at io.ktor.server.jetty.JettyKtorHandler$handle$2$invokeSuspend$$inlined$execute$1.invokeSuspend(Pipeline.kt:478)
        at io.ktor.server.jetty.JettyKtorHandler$handle$2$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
        at io.ktor.server.jetty.JettyKtorHandler$handle$2$invokeSuspend$$inlined$execute$1.invoke(Pipeline.kt)
        at io.ktor.util.debug.ContextUtilsKt.initContextInDebugMode(ContextUtils.kt:17)
        at io.ktor.server.jetty.JettyKtorHandler$handle$2.invokeSuspend(JettyKtorHandler.kt:124)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

It doesn't crash with this error though...

0
Avatar
Pavel Boger
Created

Till Rother could you clarify which image version is specified in your values.yaml for the compute service pod?

0
Avatar
Till Rother
Created

It's this one:

    pullPolicy: Always
    registry: public.registry.jetbrains.space
    repository: p/space-on-premises/docker/compute-service
    tag: 2023.1.0
0
Avatar
Pavel Boger
Created

Till Rother thanks! Am I right that it's taken from the values.yaml of a default Helm chart? Could you also check what the value is set for computeservice.worker.image.tag there? Finally, please share the computeservice part of your custom values.yaml that is supposed to be created as per the following guide:

https://www.jetbrains.com/help/space/kubernetes-installation.html#install-space-on-premises

0
Avatar
Till Rother
Created

That is the customized version, the compute-service is the default one:

computeservice:
  additionalJavaOpts: ''
  affinity:
    enabled: false
    globalAffinityKey: key1
    globalAffinityValue: value2
  autoscaling:
    enabled: true
    maxReplicas: 7
    minReplicas: 1
    targetCPU: '85'
    targetMemory: ''
  createNamespace: true
  database:
    externalSecretName: ''
    hostname: ***
    name: crcltdevdb
    password: circlet
    port: '5432'
    settings:
      idlePoolSize: '12'
    username: root
  enabled: true
  image:
    pullPolicy: Always
    registry: public.registry.jetbrains.space
    repository: p/space-on-premises/docker/compute-service
    tag: 2023.1.0
  oauth:
    clientId: space-to-computeservice-client
    clientSecret: ***
  replicas: 1
  resources:
    limits:
      cpu: "2"
      memory: 4Gi
    requests:
      cpu: 100m
      memory: 800Mi
  service:
    annotations: {}
    ports:
      internalPort: 9098
      port: 8098
  worker:
    defaultVolumeGb: 0
    extraHosts: ''
    image:
      registry: ***
      repository: p/automation-worker-test/test-containers/adva-automation-worker-k8s
      tag: latest
    namespaceSuffix: automation-jobs
    subPaths: /var/lib/docker:worker-docker
    sysbox:
      enabled: false
      podAnnotations: io.kubernetes.cri-o.userns-mode;auto:size=65536
      runtimeClassName: sysbox-runc
  workerAutoscaling:
    desiredRatioPercent: 30
    lowerLimitOfNotUsedInstances: 0
    maxScaleoutStep: 1
    scaleOutEc2InstanceLimit: 5
    upperLimitOfNotUsedInstances: 1
    workersTtl: 30m
fullnameOverride: ''
0
Avatar
Pavel Boger
Created

Till Rother, could you please explain which image is used here?

    image:
      registry: ***
      repository: p/automation-worker-test/test-containers/adva-automation-worker-k8s
      tag: latest

Is it a default 2023.1.0.684 one (from the latest chart), but stored in your private registry? Does your use case require customizing images for some reason? Thanks! 

0
Avatar
Till Rother
Created

Pavel Boger it's the default one, just with one additional ca added to the keystore in $JAVA_HOME/lib/security/cacerts.

0
Avatar
Till Rother
Created

Pavel Boger <- this one seems to be the real one
@...

There are two of you, don't know which one is the real one.

I managed to find a better solution to the custom-ca-problem of ours.
Instead of modifying the image, I added another subPath to the worker-config in values.yaml:

subPaths = "/var/lib/docker:worker-docker,<HOST-PATH-TO-CA.pem>:/var/lib/ca-certificates/pem/CA.pem"

After that it works and accepts jobs from space, handles them and reports back. Now I just need to write a job that actually reports something back, as the "Hello-World"-Jobs still report as "Unable to process script" in Space. I think that just is because they don't actually write any logs.

0
Avatar
Pavel Boger
Created

Hi Till Rother, haha, both of them are mine, but the second one is personal :)

Thanks for the info provided. I'll check this case with the team and get back to you with the updates shortly. Stay tuned!

0
Avatar
Till Rother
Created

Pavel Boger 

Alright :D

I'm still fiddling around with these subPaths, but am now suspecting that they refer to volumemounts from space-automation-worker:docker-inside-automation-worker
Seems like the certs are not properly passed through, the workers and then the container inside them are unable to connect to https://packages.*** because it uses a unknown ca.

0
Avatar
Pavel Boger
Created

Till Rother, it turned out there's a validation of the image version inside Space, therefore, the `latest` tag doesn't work in your case. Could you please republish this image with the `2023.1.0.684` tag and let me know about the results? We're going to get rid of this limitation in further versions.

0
Avatar
Till Rother
Created

@..., thanks for confirming my suspicions :D I already did so and am now facing a new issue, coming in the form of the pre-packed *.zip files in "/home/space/circlet-server-onprem/lib/plugins/pipelines/worker/" in the space-pods. I'm assuming the workers started by computeservice are downloading those from there? They contain another java install, including another instance of cacerts-keystore. I already had them modified in the beta install and am going to try that again for those ones.

 

EDIT: Didn't seem to do the trick, still getting java error in the script log:


[09:03:40]    Starting execution of 4J2ZKb2vxCvT / 35Gnw03Mcoyj / 4LEgst3gvaug step
[09:04:03]    DSL processing started
[09:04:03]    Using Automation DSL runtime version 1.2.684
[09:04:04]    Downloading DSL script content from https://space.***.com/vcs/file/automation-worker-test-test-repository-1-9f5ad80d/a97e92489ec22b24a15af640ab2d4cc2cfeadeb7//.space.kts
[09:04:04]    Cleaned up the output folder: /tmp/9154018553032511475
[09:04:04]    Exception in thread "main" runtime.net.XHRException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[09:04:04]        at runtime.net.KtorHttpKt.ktorClientRequest(KtorHttp.kt:146)
[09:04:04]        at runtime.net.KtorHttpKt.access$ktorClientRequest(KtorHttp.kt:1)
[09:04:04]        at runtime.net.KtorHttpKt$ktorClientRequest$1.invokeSuspend(KtorHttp.kt)
[09:04:04]        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
[09:04:04]        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
[09:04:04]        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
[09:04:04]        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
[09:04:04]        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
[09:04:04]        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
[09:04:04]    Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[09:04:04]        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
[09:04:04]        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
[09:04:04]        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
[09:04:04]        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
[09:04:04]        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
[09:04:04]        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
[09:04:04]        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
[09:04:04]        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
[09:04:04]        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
[09:04:04]        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
[09:04:04]        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
[09:04:04]        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
[09:04:04]        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
[09:04:04]        at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:285)
[09:04:04]        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:345)
[09:04:04]        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:523)
[09:04:04]        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
[09:04:04]        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
[09:04:04]        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
[09:04:04]        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
[09:04:04]        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
[09:04:04]        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
[09:04:04]        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
[09:04:04]        at java.base/java.lang.Thread.run(Thread.java:833)
[09:04:04]    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[09:04:04]        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
[09:04:04]        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
[09:04:04]        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
[09:04:04]        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
[09:04:04]        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
[09:04:04]        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
[09:04:04]        ... 19 more
[09:04:04]    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[09:04:04]        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
[09:04:04]        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
[09:04:04]        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
[09:04:04]        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
[09:04:04]        ... 24 more

 

0
Avatar
Pavel Boger
Created

Till Rother, could you please check if custom cacerts are included in the following image as well?

 

space:  
  automation:  
    dslCompiler:
      image:
        registry: "public.registry.jetbrains.space"
        repository: "p/space/containers/space-automation-dsl-compile"
        tag: "2023.1.0.684"
0
Avatar
Till Rother
Created

Hi Pavel Boger

I didn't modify any of the other images, but will try that next, thanks! Will report back :)

0
Avatar
Till Rother
Created

@...

I was finally able to test your suggestion and the computeservice + workers now work as expected! Thank you for your continuous support!

0

Please sign in to leave a comment.