exception from automation worker - "runtime.net.XHRException: Connection is closed"
(did helm update/upgrade prior)
attempting to run simple automation script:
Exception:
[23:59:38] Starting execution of Sj7nK21PW9S / 23ViqH0p9y7k / 3YTjIh3C8exF step
[23:59:43] ERROR: Couldn't log in to docker registry https://space-packages.mydomain.com. Exit code: 1. Details: WARNING! Using --password via the CLI is insecure. Use --password-stdin.
[23:59:43] Error response from daemon: Get "https://space-packages.mydomain.com/v2/": EOF
[23:59:43]
[00:00:03] DSL processing started
[00:00:03] Using Automation DSL runtime version 1.2.684
[00:00:04] Downloading DSL script content from https://space-portal.mydomain.com/vcs/file/microservices-8f4fee9e/de38e7d48c63b626bbe1e0937db7125919c9f5bb//.space.kts
[00:00:10] Cleaned up the output folder: /tmp/7164266433534159908
[00:00:10] Exception in thread "main" runtime.net.XHRException: Connection is closed
[00:00:10] at runtime.net.KtorHttpKt.ktorClientRequest(KtorHttp.kt:146)
[00:00:10] at runtime.net.KtorHttpKt.access$ktorClientRequest(KtorHttp.kt:1)
[00:00:10] at runtime.net.KtorHttpKt$ktorClientRequest$1.invokeSuspend(KtorHttp.kt)
[00:00:10] at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
[00:00:10] at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
[00:00:10] at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:570)
[00:00:10] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
[00:00:10] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:677)
[00:00:10] at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:664)
[00:00:10] Caused by: org.apache.http.ConnectionClosedException: Connection is closed
[00:00:10] at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:356)
[00:00:10] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
[00:00:10] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
[00:00:10] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
[00:00:10] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
[00:00:10] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
[00:00:10] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
[00:00:10] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
[00:00:10] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
[00:00:10] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
[00:00:10] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
[00:00:10] at java.base/java.lang.Thread.run(Thread.java:833)
Clearly Im missing something in my values.yaml.. where to look?
Please sign in to leave a comment.
Also this exception automation pod:
00:00:21.239 ERROR
io.ktor.client.plugins.ClientRequestException: Client request(GET http://jb-space-computeservice.jb-space:9098/pipelines/worker/version) invalid: 401 Unauthorized. Text: ""
at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:51)
at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
at io.ktor.client.plugins.HttpCallValidator.validateResponse(HttpCallValidator.kt:47)
at io.ktor.client.plugins.HttpCallValidator.access$validateResponse(HttpCallValidator.kt:40)
at io.ktor.client.plugins.HttpCallValidator$Companion$install$3.invokeSuspend(HttpCallValidator.kt:148)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
at circlet.pipelines.updater.UpdaterMainKt$runUpdater$1.invokeSuspend$lambda$4(UpdaterMain.kt:120)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Oddly - it seems to be getting a proper git clone as it reported a bug in the script initially
Ethrbunny, one of the possible reasons is that https://space-portal.mydomain.com uses self-signed certificates which are not validated by cacerts from within the automation containers. Could you please check if that's the case?
No - Im getting the certs from ACM. Its all legit now.
I simplified the .kts file to just run 'make test'. The jobs are completing but only insomuch as they are unable to clone the repo.
For some reason after making one .kts file change I have 4 automation pods started.. but only one is able to run apparently. The other 3 are stuck in 'pending'. Is this a limitation of the 'free' on-prem installation or something I've set in values.yaml?
13:47:37.635 INFO [WORKER_LOG_LINE@2023.1.1.2496] WARN c.p.b.BootstrapCommandsProcessor - Problem during git operation │
│ 13:47:37.635 INFO [WORKER_LOG_LINE@2023.1.1.2496] circlet.pipelines.bootstrap.GitCommandProcessingException: Git process exited with non-zero exit code [128]: ssh: connect to host space-git.mydomain.com │
│ 13:47:37.635 INFO [WORKER_LOG_LINE@2023.1.1.2496] fatal: Could not read from remote repository. │
│ 13:47:37.635 INFO [WORKER_LOG_LINE@2023.1.1.2496] │
│ 13:47:37.635 INFO [WORKER_LOG_LINE@2023.1.1.2496] Please make sure you have the correct access rights │
│ 13:47:37.635 INFO [WORKER_LOG_LINE@2023.1.1.2496] and the repository exists. │
│ 13:47:37.636 INFO [WORKER_LOG_LINE@2023.1.1.2496] │
Perhaps related to ssh unavailability in my other ticket?
Thought maybe perhaps setting "vcs.secrets.sshServerKey" would help but I can't figure out what syntax it wants. Have tried a variety of rsa keys in various formats. All give me "java.lang.IllegalStateException: Cannot parse provided private key"
Ethrbunny, format is raw RSA 4096 key generated by openssh, e.g. in values.yaml:
vcs:secrets:sshServerKey: |-----BEGIN PRIVATE KEY-----MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQDNq0SvTXW6YefQD8Mbp0yYngZypjiluqr3O0dW4JmvQT5sTF4Zq9Exrq+uk4153HWcAN1l372etA2D........-----END PRIVATE KEY-----Sample command for generate is
openssl genrsa 4096 > key.pem