ON-PREM: Login failure: javax.crypto.AEADBadTagException: Tag mismatch!
Recently one of our test installs started failing all logins - via browser as well as app - with a message of:
{"error":"internal-server-error","error_description":"Internal Server Error"}
Monitoring the log at the same time gives:
Jan 02 11:14:19 services-space space[3081]: 2023-01-02 10:14:19.867 [ktor-jetty-8084-9084-6] ERROR Application [trace_id=7183996199161761031] - Server err
or
Jan 02 11:14:19 services-space space[3081]: javax.crypto.AEADBadTagException: Tag mismatch!
Jan 02 11:14:19 services-space space[3081]: at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
Jan 02 11:14:19 services-space space[3081]: at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
Jan 02 11:14:19 services-space space[3081]: at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
Jan 02 11:14:19 services-space space[3081]: at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
Jan 02 11:14:19 services-space space[3081]: at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
Jan 02 11:14:19 services-space space[3081]: at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.ar.AESEncryptionUtils.b(AESEncryptionUtils.java:27)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.an.c.l.ETwoFactorAuthentication.a(ETwoFactorAuthentication.java:11)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.an.c.l.ETwoFactorAuthentication.a(ETwoFactorAuthentication.java:48)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.an.c.l.c.a(c.java:58)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.an.c.l.g$b$c$a$d.a(g$b$c$a$d.java:5)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.an.c.l.g$b$c$a$d.invoke(g$b$c$a$d.java:2)
Jan 02 11:14:19 services-space space[3081]: at circlet.server.db.OrgDbContext$tx$2.invoke(OrgDbContext.kt:2)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$c$a$b$d.a(bd$c$a$b$d.java:2)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$c$a$b$d.invoke(bd$c$a$b$d.java:1)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$k$a.invoke(bd$k$a.java:35)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.m.ab.a(ab.java:50)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$k.a(bd$k.java:28)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$k.invoke(bd$k.java:25)
Jan 02 11:14:19 services-space space[3081]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction$run(Th
readLocalTransactionManager.kt:198)
Jan 02 11:14:19 services-space space[3081]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.access$inTopLevelTransaction
$run(ThreadLocalTransactionManager.kt:1)
Jan 02 11:14:19 services-space space[3081]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt$inTopLevelTransaction$1.invo
ke(ThreadLocalTransactionManager.kt:221)
Jan 02 11:14:19 services-space space[3081]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.keepAndRestoreTransactionRef
AfterRun(ThreadLocalTransactionManager.kt:229)
Jan 02 11:14:19 services-space space[3081]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction(Thread
LocalTransactionManager.kt:220)
Jan 02 11:14:19 services-space space[3081]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction$defaul
t(ThreadLocalTransactionManager.kt:179)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd.a(bd.java:100)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd.b(bd.java:224)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$c$a$b.a(bd$c$a$b.java:1)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$c$a$b.invoke(bd$c$a$b.java:5)
Jan 02 11:14:19 services-space space[3081]: at libraries.coroutines.extra.CoroutineSessionElementKt.withMetricsSessionScopeBlocking(CoroutineSessi
onElement.kt:23)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$c$a.a(bd$c$a.java:34)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.bd$c$a.invoke(bd$c$a.java:16)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.TxPool$a$c.a(TxPool$a$c.java:2)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.TxPool$a$c.invoke(TxPool$a$c.java:1)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.TxPool$b.a(TxPool$b.java:66)
Jan 02 11:14:19 services-space podman-space-start[1000]: 2023-01-02 10:14:19.867 [ktor-jetty-8084-9084-6] ERROR Application [trace_id=7183996199161761031]
- Server error
Jan 02 11:14:19 services-space podman-space-start[1000]: javax.crypto.AEADBadTagException: Tag mismatch!
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.jav
a:623)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2202)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.ar.AESEncryptionUtils.b(AESEncryptionUtils.java:27)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.an.c.l.ETwoFactorAuthentication.a(ETwoFactorAuthentication.java:11)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.an.c.l.ETwoFactorAuthentication.a(ETwoFactorAuthentication.java:48)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.an.c.l.c.a(c.java:58)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.an.c.l.g$b$c$a$d.a(g$b$c$a$d.java:5)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.an.c.l.g$b$c$a$d.invoke(g$b$c$a$d.java:2)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.server.db.OrgDbContext$tx$2.invoke(OrgDbContext.kt:2)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$c$a$b$d.a(bd$c$a$b$d.java:2)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$c$a$b$d.invoke(bd$c$a$b$d.java:1)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$k$a.invoke(bd$k$a.java:35)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.m.ab.a(ab.java:50)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$k.a(bd$k.java:28)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$k.invoke(bd$k.java:25)
Jan 02 11:14:19 services-space podman-space-start[1000]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTrans
action$run(ThreadLocalTransactionManager.kt:198)
Jan 02 11:14:19 services-space podman-space-start[1000]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.access$inTopLev
elTransaction$run(ThreadLocalTransactionManager.kt:1)
Jan 02 11:14:19 services-space podman-space-start[1000]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt$inTopLevelTrans
action$1.invoke(ThreadLocalTransactionManager.kt:221)
Jan 02 11:14:19 services-space podman-space-start[1000]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.keepAndRestoreT
ransactionRefAfterRun(ThreadLocalTransactionManager.kt:229)
Jan 02 11:14:19 services-space podman-space-start[1000]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTrans
action(ThreadLocalTransactionManager.kt:220)
Jan 02 11:14:19 services-space podman-space-start[1000]: at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTrans
action$default(ThreadLocalTransactionManager.kt:179)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd.a(bd.java:100)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd.b(bd.java:224)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$c$a$b.a(bd$c$a$b.java:1)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$c$a$b.invoke(bd$c$a$b.java:5)
Jan 02 11:14:19 services-space podman-space-start[1000]: at libraries.coroutines.extra.CoroutineSessionElementKt.withMetricsSessionScopeBlocking(C
oroutineSessionElement.kt:23)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$c$a.a(bd$c$a.java:34)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.bd$c$a.invoke(bd$c$a.java:16)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.TxPool$a$c.a(TxPool$a$c.java:2)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.TxPool$a$c.invoke(TxPool$a$c.java:1)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.TxPool$b.a(TxPool$b.java:66)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.TxPool$b.a(TxPool$b.java:12)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.TxPool$b$c.invokeSuspend(TxPool$b$c.java:2)
Jan 02 11:14:19 services-space podman-space-start[1000]: at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
Jan 02 11:14:19 services-space podman-space-start[1000]: at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
Jan 02 11:14:19 services-space podman-space-start[1000]: at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:279)
Jan 02 11:14:19 services-space podman-space-start[1000]: at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
Jan 02 11:14:19 services-space podman-space-start[1000]: at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
Jan 02 11:14:19 services-space podman-space-start[1000]: at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
Jan 02 11:14:19 services-space podman-space-start[1000]: at circlet.platform.a.l.TxPool$b.a(TxPool$b.java:3)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
28)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
28)
Jan 02 11:14:19 services-space podman-space-start[1000]: at java.base/java.lang.Thread.run(Thread.java:829)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.TxPool$b.a(TxPool$b.java:12)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.TxPool$b$c.invokeSuspend(TxPool$b$c.java:2)
Jan 02 11:14:19 services-space space[3081]: at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
Jan 02 11:14:19 services-space space[3081]: at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
Jan 02 11:14:19 services-space space[3081]: at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:279)
Jan 02 11:14:19 services-space space[3081]: at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
Jan 02 11:14:19 services-space space[3081]: at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
Jan 02 11:14:19 services-space space[3081]: at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
Jan 02 11:14:19 services-space space[3081]: at circlet.platform.a.l.TxPool$b.a(TxPool$b.java:3)
Jan 02 11:14:19 services-space space[3081]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Jan 02 11:14:19 services-space space[3081]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Jan 02 11:14:19 services-space space[3081]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Jan 02 11:14:19 services-space space[3081]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jan 02 11:14:19 services-space space[3081]: at java.base/java.lang.Thread.run(Thread.java:829)
Obviously this halts our test dead in its tracks. I can't immediately identify what makes this install different from the install which continues to work. Where would I start investigating this?
Please sign in to leave a comment.
This seems the closest fit to identifying the underlying issue, but I'm obviously not familiar with the implementation here: https://www.ibm.com/support/pages/failure-javaxcryptoaeadbadtagexception-due-defect-java-8-tlsv13-implementation
Emil could it be possible that the DB of this installation has been migrated to a new installation with the new set of secrets in config files? This exception may appear in case the DB was initially created for one masterSecret, and then switched to a new one.
Please also provide more information about your installation type (PoC or Kubernetes), any changes in configuration, etc? Thanks!
Hey @... thank you for the follow-up! I have not explicitly performed a database migration, no, but if we can salvage access that way then I am certainly up for it. The install is PoC, running in Podman on NixOS. You can see the exact changes we make to the default config in the nix config here (setting URLs for base, frontend, and circlet as well as configuring email). As mentioned we have a couple of other installs with identical configuration which continue to operate just fine.
Hey Pavel Boger. I just realized I may well have iterated a bit on the config here, which would have re-run the config initialisation referenced above - thus resetting the master secret as you point out. I will dig through backups to confirm and let you know. Afterwards I'll share an updated version of our Nix config doing an initial extraction of the generated key and then re-populating it into the config file if an already extracted key is found.
That does indeed seem to have done the trick. I'll ping here when the Nix config change has been pushed.
Separately: It appears that a repository was added while new keys were in effect and as a result we are not able to delete it via the web interface - with keys now restored. Is there a separate way to delete this repository so we may re-create it with the correct keys?
For anyone reading:
space.on-premises.confholds copies of keys which need synchronising tovcs.on-premises.properties