[On-Prem / Docker] Cannot consume nuget packages, getting redirect errors
I have setup spaces using Docker and I setup a nuget package feed. I use teamcity as my CI/CD pipeline. Using teamcity I am able to push nuget packages to the feed, but when I try to add them to a project I get the following errors
Failed to download package 'Calendar.Api.0.0.0.41' from 'https://packages.examplesoft.com/nuget/p/exampledev/example-nuget/v3/flatcontainer/calendar.api/0.0.0.41/calendar.api.0.0.0.41.nupkg'.
Response status code does not indicate success: 307 (Temporary Redirect).
Enhanced retry: HttpRetryHandler is in a state that retry would have been abandoned or not waited if it were not enabled.
GET https://packages.examplesoft.com/nuget/p/exampledev/example-nuget/v3/flatcontainer/calendar.api/0.0.0.41/calendar.api.0.0.0.41.nupkg
TemporaryRedirect https://packages.examplesoft.com/nuget/p/exampledev/example-nuget/v3/flatcontainer/calendar.api/0.0.0.41/calendar.api.0.0.0.41.nupkg 38ms
Failed to download package 'Calendar.Api.0.0.0.41' from 'https://packages.examplesoft.com/nuget/p/exampledev/example-nuget/v3/flatcontainer/calendar.api/0.0.0.41/calendar.api.0.0.0.41.nupkg'.
Response status code does not indicate success: 307 (Temporary Redirect).
Also I can see the package in spaces web ui. However, if I click on the three dots and select download, I get the following error
Mixed Content: The page at 'https://spaces.examplesoft.com/p/exampledev/packages/nuget/example-nuget/Calendar.Api?v=0.0.0.41&tab=overview' was loaded over HTTPS, but requested an insecure resource 'http://minio:9000/packages-local/7120701354309526791/nuget/7032838529304953598/flatcontainer/calendar.api/0.0.0.41/calendar.api.0.0.0.41.nupkg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230706T123533Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=space-access-key%2F20230706%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=0b4f96840c0748f2d24cf2be40b133de618c479e55e63cbe29bd5ca135018cac'. This request has been blocked; the content must be served over HTTPS.
index.fafdb698def9d27a88cf.js:1 209449ms [platform-runtime/CoroutineExceptionHandler.kt]: Unhandled exception in unnamed coroutine TypeError: Failed to fetch
at zp.doResume (index.fafdb698def9d27a88cf.js:1:17526492)
at hR.download_w9w408$ (index.fafdb698def9d27a88cf.js:1:17807880)
at de.doResume (5269.f82b02b1c90fd1449c26.js:1:44504)
at he.downloadPackageVersion_0 (5269.f82b02b1c90fd1449c26.js:1:44931)
at $e.doResume (5269.f82b02b1c90fd1449c26.js:1:43355)
at vzt.action (5269.f82b02b1c90fd1449c26.js:1:44121)
at mv.doResume (index.fafdb698def9d27a88cf.js:1:17624476)
at Ri.resumeWith_tl1gpc$ (index.fafdb698def9d27a88cf.js:1:422364)
at ea.run (index.fafdb698def9d27a88cf.js:1:942098)
at hl.process (index.fafdb698def9d27a88cf.js:1:989633)
at index.fafdb698def9d27a88cf.js:1:989210
Please sign in to leave a comment.
Noamalper, thanks for reporting this. It's a known issue that the Packages service is referring to the internal minio hostname, instead of a public one. Please upgrade the installation to the latest 2023.2.0 version and update the configuration as it's described in Step 7 of this guide.
That seems to have fixed the issue. I can now publish and consumer. However, in the new version I cannot browse the repository. When I go to the repository main page all I see is
Even though the repository contains a bunch of packages.
Noamalper let's check the logs from the browser console, as well as the logs from the packages and space containers. Could you also check any other data like Git repos in your project? Is it still accessible?
First thank you so much for your help I really appreciate it. I can access my git repos. Here are the logs from my webconsole, packages and space
================webconsole=====================
Navigated to https://spaces.examplesoft.com/p/exampledev/packages/nuget/example-nuget
index.7dfb32777a5d7a474c84.js:1 3968ms [LoadingValue.kt]: Z_: Call to ProjectPackages/getRepositoryStats failed with error: Unable to get repository example-nuget stats
at Ft.ta (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11521075)
at Ft.ta (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11516435)
at Ft.fp6 (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11516035)
at tn.h [as i4r_1] (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11374782)
at b.ta (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:10833470)
at Lf.wa (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14299482)
at Lf.a7 (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14299724)
at Lt.i13 (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14652964)
at Lt.i1v (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14661241)
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14661956
b.f4k @ index.7dfb32777a5d7a474c84.js:1
b.warn_t__log_replace__ @ index.7dfb32777a5d7a474c84.js:1
we.ta @ index.7dfb32777a5d7a474c84.js:1
Lf.wa @ index.7dfb32777a5d7a474c84.js:1
Lf.a7 @ index.7dfb32777a5d7a474c84.js:1
Lt.i13 @ index.7dfb32777a5d7a474c84.js:1
Lt.i1v @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
Promise.then (async)
Lt.j1v @ index.7dfb32777a5d7a474c84.js:1
Lt.x1v @ index.7dfb32777a5d7a474c84.js:1
Lt.f14 @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
br @ index.7dfb32777a5d7a474c84.js:1
Lt.ty @ index.7dfb32777a5d7a474c84.js:1
Bu @ index.7dfb32777a5d7a474c84.js:1
Lt.g1c @ index.7dfb32777a5d7a474c84.js:1
vu @ index.7dfb32777a5d7a474c84.js:1
cu @ index.7dfb32777a5d7a474c84.js:1
Lt.n1e @ index.7dfb32777a5d7a474c84.js:1
Ft.c9w @ index.7dfb32777a5d7a474c84.js:1
xP.c9w @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
index.7dfb32777a5d7a474c84.js:1 3970ms [AppRouting.kt]: Routing to spaces.examplesoft.com/p/exampledev/packages/nuget/example-nuget (NuGetDescriptor) complete in 1849 ms
index.7dfb32777a5d7a474c84.js:1 5096ms [LoadingValue.kt]: Zo: Job was cancelled
at Lt.rw (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14614519)
at ps.Vi [as sw] (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:14543451)
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:10827845
at b.n4p (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:10830227)
at Rt.a (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:10827302)
at b.n4p (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:10830227)
at ld (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11843203)
at we.q (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11960011)
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11844226
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:11943362
at nf.e [as q2c2_1] (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12186526)
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12207057
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12180498
at _n.q2bi (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12180594)
at _n.y2c2 (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12207021)
at tf (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12102732)
at _n.b2c3 (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12207420)
at _n.z2c2 (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12208012)
at https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12102521
at Ql (https://spaces.examplesoft.com/static/index.7dfb32777a5d7a474c84.js:1:12101960)
b.f4k @ index.7dfb32777a5d7a474c84.js:1
b.warn_t__log_replace__ @ index.7dfb32777a5d7a474c84.js:1
we.ta @ index.7dfb32777a5d7a474c84.js:1
Lf.wa @ index.7dfb32777a5d7a474c84.js:1
Lf.a7 @ index.7dfb32777a5d7a474c84.js:1
Lt.i13 @ index.7dfb32777a5d7a474c84.js:1
Lt.i1v @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
Promise.then (async)
Lt.j1v @ index.7dfb32777a5d7a474c84.js:1
Lt.x1v @ index.7dfb32777a5d7a474c84.js:1
Lt.f14 @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
br @ index.7dfb32777a5d7a474c84.js:1
Lt.ty @ index.7dfb32777a5d7a474c84.js:1
Bu @ index.7dfb32777a5d7a474c84.js:1
Lt.g1c @ index.7dfb32777a5d7a474c84.js:1
vu @ index.7dfb32777a5d7a474c84.js:1
cu @ index.7dfb32777a5d7a474c84.js:1
Lt.n1e @ index.7dfb32777a5d7a474c84.js:1
Ft.c9w @ index.7dfb32777a5d7a474c84.js:1
xP.c9w @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
index.7dfb32777a5d7a474c84.js:1 5098ms [kclient/WebSocketTransport.kt]: no handler with id = 84 found
b.warn_l__log_replace__ @ index.7dfb32777a5d7a474c84.js:1
Ft.ta @ index.7dfb32777a5d7a474c84.js:1
Ft.ta @ index.7dfb32777a5d7a474c84.js:1
Ft.fp6 @ index.7dfb32777a5d7a474c84.js:1
h @ index.7dfb32777a5d7a474c84.js:1
b.ta @ index.7dfb32777a5d7a474c84.js:1
Lf.wa @ index.7dfb32777a5d7a474c84.js:1
Lf.a7 @ index.7dfb32777a5d7a474c84.js:1
Lt.i13 @ index.7dfb32777a5d7a474c84.js:1
Lt.i1v @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
Promise.then (async)
Lt.j1v @ index.7dfb32777a5d7a474c84.js:1
Lt.x1v @ index.7dfb32777a5d7a474c84.js:1
Lt.f14 @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
br @ index.7dfb32777a5d7a474c84.js:1
Lt.ty @ index.7dfb32777a5d7a474c84.js:1
Bu @ index.7dfb32777a5d7a474c84.js:1
Lt.g1c @ index.7dfb32777a5d7a474c84.js:1
vu @ index.7dfb32777a5d7a474c84.js:1
cu @ index.7dfb32777a5d7a474c84.js:1
Lt.n1e @ index.7dfb32777a5d7a474c84.js:1
Ft.c9w @ index.7dfb32777a5d7a474c84.js:1
xP.c9w @ index.7dfb32777a5d7a474c84.js:1
(anonymous) @ index.7dfb32777a5d7a474c84.js:1
============spaces========================
2023-07-07 12:33:14.336 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'description' of function Boards.updateBoard
2023-07-07 12:33:14.336 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'swimlaneKey' of function Boards.updateBoard
2023-07-07 12:33:14.337 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'description' of function Boards.updateSprint
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'name' of function Checklists.editPersonalChecklist
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'description' of function Checklists.editPersonalChecklist
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'name' of function Checklists.editProjectChecklist
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'description' of function Checklists.editProjectChecklist
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'owner' of function Checklists.editProjectChecklist
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'tag' of function Checklists.editProjectChecklist
2023-07-07 12:33:14.338 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'topics' of function Checklists.editProjectChecklist
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'itemText' of function Checklists.updatePlanItem
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'itemDone' of function Checklists.updatePlanItem
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'title' of function Issues.editIssue
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'description' of function Issues.editIssue
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'assignee' of function Issues.editIssue
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'status' of function Issues.editIssue
2023-07-07 12:33:14.341 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'dueDate' of function Issues.editIssue
2023-07-07 12:33:14.343 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'assigneeMissingPolicy' of function Issues.importIssues
2023-07-07 12:33:14.343 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'statusMissingPolicy' of function Issues.importIssues
2023-07-07 12:33:14.344 [api call executor 19] WARN c.p.api.httpApi.HttpApiParameters [trace_id=7253053835628669587, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=13, connectionId=9ak3s95da3] - Failed to resolve expression '_' as default value of parameter 'onExistsPolicy' of function Issues.importIssues
2023-07-07 12:33:14.385 [[tx-pool] shared-20] INFO c.s.services.calendar.HolidaysCache [trace_id=7253053835728429763, orgDomain=space, txOriginName=Cache holidaysCache calculation, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=17, connectionId=9ak3s95da3] - starting HolidaysCache update
2023-07-07 12:33:14.387 [[tx-pool] shared-20] INFO c.s.services.calendar.HolidaysCache [trace_id=7253053835728429763, orgDomain=space, txOriginName=Cache holidaysCache calculation, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=17, connectionId=9ak3s95da3] - 0 holidays are loaded
2023-07-07 12:33:14.553 [ktor-server-jetty-198] INFO c.platform.a.a.PlatformApplication [dbType=postgresql, trace_id=7253053839901945852, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - [c5mx1lqsz6, user:admin/98.117.198.156]: Incoming connection.
2023-07-07 12:33:14.712 [api call executor 117] INFO c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053839933251021, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=36, connectionId=9ak3s95da3] - Forbidden refs: tgl-ord-statevzTqH1QVVy9/top-level-page-extension-not found
2023-07-07 12:33:14.947 [api call executor 41] INFO c.s.s.p.ResponsibilityProvider [trace_id=7253053839951521857, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=55, connectionId=9ak3s95da3] - calling default responsibility
2023-07-07 12:33:15.325 [api call executor 20] INFO c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053844307243134, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=85, connectionId=9ak3s95da3] - Forbidden refs: profile-project-sidebar-settings/1uNx061C68mm-not found,tgl-ord-statevzTqH1QVVy9/project-features_1uNx061C68mm-not found
2023-07-07 12:33:22.314 [api call executor 101] WARN c.s.p.VersionedPackageRepositoryClientBase [trace_id=7253053874262999629, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=115, connectionId=9ak3s95da3] - Unable to list nuget packages in repository example-nuget: Client request(GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/pkgs?query=&size=50) invalid: 404 Not Found. Text: ""
2023-07-07 12:33:22.315 [api call executor 101] WARN circlet.platform.a.ar.g [trace_id=7253053874262999629, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=115, connectionId=9ak3s95da3] - 9ak3s95da3 115: Client request failed with exception ProjectPackages/listPackages
circlet.platform.a.aw.s: Unable to list nuget packages
at circlet.server.packages.VersionedPackageRepositoryClientBase.listPackages$suspendImpl(VersionedPackageRepositoryClientBase.kt:464)
at circlet.server.packages.VersionedPackageRepositoryClientBase$listPackages$1.invokeSuspend(VersionedPackageRepositoryClientBase.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-07-07 12:33:22.315 [api call executor 19] WARN c.s.p.PackageRepositoryClientBase [trace_id=7253053874473450629, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=114, connectionId=9ak3s95da3] - Unable to get repository example-nuget stats
io.ktor.client.plugins.ClientRequestException: Client request(GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/stats) invalid: 404 Not Found. Text: ""
at runtime.utils.BackoffKt$getResponse$3$2.invokeSuspend(backoff.kt:35)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-07-07 12:33:22.316 [api call executor 19] WARN circlet.platform.a.ar.g [trace_id=7253053874473450629, orgDomain=space, ip=98.117.198.156, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=114, connectionId=9ak3s95da3] - 9ak3s95da3 114: Client request failed with exception ProjectPackages/getRepositoryStats
circlet.platform.a.aw.s: Unable to get repository example-nuget stats
at circlet.server.packages.PackageRepositoryClientBase.getRepositoryStats$suspendImpl(PackageRepositoryClientBase.kt:261)
at circlet.server.packages.PackageRepositoryClientBase$getRepositoryStats$1.invokeSuspend(PackageRepositoryClientBase.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-07-07 12:33:24.542 [redisson-3-5] INFO circlet.platform.a.k.h [] - Node appeared: ClusterNode(id=99bde136aa3d, app=Packages, version=2023.2.0.128)
2023-07-07 12:33:46.765 [ktor-server-jetty-222] INFO c.platform.a.a.PlatformApplication [dbType=postgresql, trace_id=7253053977464215249, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - [ab7hacxki3, user:admin/192.168.1.1]: Incoming connection.
2023-07-07 12:33:46.773 [ktor-server-jetty-225] INFO c.platform.a.a.PlatformApplication [dbType=postgresql, trace_id=7253053977505090869, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - [6qkb1l782n, user:admin/98.117.198.156]: Incoming connection.
2023-07-07 12:33:46.783 [api call executor 105] WARN c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053977342908570, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=176, connectionId=ab7hacxki3] - principal admin has no access to 1 arenas in ChatUnreadStatusArenaProvider, first 1 is [m2-status2RNglp3zztcm]
2023-07-07 12:33:46.783 [api call executor 98] WARN c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053977342908570, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=176, connectionId=ab7hacxki3] - principal admin has no access to 1 arenas in ChatContactGroupOrderArenaProvider, first 1 is [m2-group-order2RNglp3zztcm]
2023-07-07 12:33:46.783 [api call executor 109] WARN c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053977342908570, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=176, connectionId=ab7hacxki3] - principal admin has no access to 1 arenas in ChatContactsGroupsArenaProvider, first 1 is [m2-groups2RNglp3zztcm]
2023-07-07 12:33:46.784 [api call executor 113] WARN c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053977342908570, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=176, connectionId=ab7hacxki3] - principal admin has no access to 1 arenas in ToggleableOrderableStateArenaProvider, first 1 is [tgl-ord-state2RNglp3zztcm]
2023-07-07 12:33:46.783 [api call executor 106] WARN c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053977342908570, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=176, connectionId=ab7hacxki3] - principal admin has no access to 1 arenas in ChatContactsArenaProvider, first 1 is [m2-contacts2RNglp3zztcm]
2023-07-07 12:33:46.784 [api call executor 21] WARN c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253053977342908570, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=176, connectionId=ab7hacxki3] - principal admin has no access to 1 arenas in NavBarProjectsArenaProvider, first 1 is [navbar-projects_2RNglp3zztcm]
2023-07-07 12:35:02.922 [ktor-server-jetty-225] INFO circlet.platform.a.ar.g [dbType=postgresql, trace_id=7253053835755571348, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - [9ak3s95da3, user:admin/98.117.198.156] terminating connection lifetime
2023-07-07 12:35:02.924 [ktor-server-jetty-225] INFO circlet.platform.a.ar.g [dbType=postgresql, trace_id=7253053835755571348, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - [9ak3s95da3, user:admin/98.117.198.156] Connection closed
2023-07-07 12:35:03.287 [ktor-server-jetty-197] INFO c.platform.a.a.PlatformApplication [dbType=postgresql, trace_id=7253054303801604185, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - [50vnadolzd, user:admin/192.168.1.1]: Incoming connection.
2023-07-07 12:35:03.391 [[tx-pool] shared-10] INFO c.s.services.channels.PeerService [trace_id=7253054308162082872, orgDomain=space, txOriginName=M2/channel, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=14, connectionId=50vnadolzd] - resolveChannel found no channels for key = #Blog
2023-07-07 12:35:03.612 [api call executor 95] WARN c.s.p.PackageRepositoryClientBase [trace_id=7253054308283762447, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=35, connectionId=50vnadolzd] - Unable to get repository example-nuget stats
io.ktor.client.plugins.ClientRequestException: Client request(GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/stats) invalid: 404 Not Found. Text: ""
at runtime.utils.BackoffKt$getResponse$3$2.invokeSuspend(backoff.kt:35)
at runtime.utils.BackoffKt$getResponse$3$2.invoke(backoff.kt)
at runtime.utils.BackoffKt$getResponse$3$2.invoke(backoff.kt)
at io.ktor.client.statement.HttpStatement.execute(HttpStatement.kt:50)
at io.ktor.client.statement.HttpStatement$execute$1.invokeSuspend(HttpStatement.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-07-07 12:35:03.612 [api call executor 95] WARN circlet.platform.a.ar.g [trace_id=7253054308283762447, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=35, connectionId=50vnadolzd] - 50vnadolzd 35: Client request failed with exception ProjectPackages/getRepositoryStats
circlet.platform.a.aw.s: Unable to get repository example-nuget stats
at circlet.server.packages.PackageRepositoryClientBase.getRepositoryStats$suspendImpl(PackageRepositoryClientBase.kt:261)
at circlet.server.packages.PackageRepositoryClientBase$getRepositoryStats$1.invokeSuspend(PackageRepositoryClientBase.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-07-07 12:35:03.615 [api call executor 96] WARN c.s.p.VersionedPackageRepositoryClientBase [trace_id=7253054308179484604, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=36, connectionId=50vnadolzd] - Unable to list nuget packages in repository example-nuget: Client request(GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/pkgs?query=&size=50) invalid: 404 Not Found. Text: ""
2023-07-07 12:35:03.615 [api call executor 96] WARN circlet.platform.a.ar.g [trace_id=7253054308179484604, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=36, connectionId=50vnadolzd] - 50vnadolzd 36: Client request failed with exception ProjectPackages/listPackages
circlet.platform.a.aw.s: Unable to list nuget packages
at circlet.server.packages.VersionedPackageRepositoryClientBase.listPackages$suspendImpl(VersionedPackageRepositoryClientBase.kt:464)
at circlet.server.packages.VersionedPackageRepositoryClientBase$listPackages$1.invokeSuspend(VersionedPackageRepositoryClientBase.kt)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-07-07 12:35:03.918 [api call executor 51] INFO c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253054308306354449, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=67, connectionId=50vnadolzd] - Forbidden refs: tgl-ord-statevzTqH1QVVy9/top-level-page-extension-not found
2023-07-07 12:35:03.952 [api call executor 76] INFO c.platform.a.ar.a.c.RefServiceImpl [trace_id=7253054308132157631, orgDomain=space, ip=192.168.1.1, dbType=postgresql, clientTimezone=America/New_York, principalName=admin, principalId=7778570276218047978, userId=9142502202025435541, orgId=1243897838375487330, dbKey=postgres, clientType=Browser, requestId=77, connectionId=50vnadolzd] - Forbidden refs: profile-project-sidebar-settings/1uNx061C68mm-not found,tgl-ord-statevzTqH1QVVy9/project-features_1uNx061C68mm-not found
2023-07-07 12:35:35.077 [server dispatch thread 8] INFO c.s.s.spaceNews.SpaceNewsAppCache [dbType=postgresql, trace_id=7253053827223523395, orgDomain=space, orgId=1243897838375487330, dbKey=postgres] - applyUpdates: cache size = 0
==============packages===================
[2023-07-07 12:33:04,084] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - isolateInternalQueries..........false
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - jdbcUrl.........................jdbc:postgresql://postgres:5432/spacedb
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - keepaliveTime...................0
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - leakDetectionThreshold..........300000
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - maxLifetime.....................1800000
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - maximumPoolSize.................96
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - metricRegistry..................none
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - metricsTrackerFactory...........none
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - minimumIdle.....................12
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - password........................<masked>
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - poolName........................"HikariPool-1"
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - readOnly........................false
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - registerMbeans..................false
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - scheduledExecutor...............none
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - schema..........................none
[2023-07-07 12:33:04,085] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - threadFactory...................internal
[2023-07-07 12:33:04,086] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - transactionIsolation............"TRANSACTION_REPEATABLE_READ"
[2023-07-07 12:33:04,086] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - username........................"space"
[2023-07-07 12:33:04,086] DEBUG [-pool] shared-2] com.zaxxer.hikari.HikariConfig [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - validationTimeout...............5000
[2023-07-07 12:33:04,086] INFO [-pool] shared-2] zaxxer.hikari.HikariDataSource [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - HikariPool-1 - Starting...
[2023-07-07 12:33:04,167] INFO [-pool] shared-2] .zaxxer.hikari.pool.HikariPool [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@4b5e5af
[2023-07-07 12:33:04,167] INFO [-pool] shared-2] zaxxer.hikari.HikariDataSource [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - HikariPool-1 - Start completed.
[2023-07-07 12:33:04,189] DEBUG [spatch thread 3] circlet.platform.a.l.r [dbType=postgresql, trace_id=7253053797152527670, dbKey=postgres] - Create schema 'crclt_packages_meta' if not exists took 0.122s
[2023-07-07 12:33:04,196] INFO [spatch thread 3] circlet.platform.a.l.r [dbType=postgresql, dbKey=postgres] - Migrations which were disabled from supervisor are : []
[2023-07-07 12:33:04,220] INFO [-pool] shared-3] circlet.platform.a.l.c.e [dbType=postgresql, trace_id=7253053797189464882, txOriginName=ApplicationLifecycle:startServices:circlet.packages.server.db.PackagesMetaDBConnection:create locks table and check schema, dbKey=postgres] - Loading version for db
[2023-07-07 12:33:04,221] INFO [-pool] shared-3] circlet.platform.a.l.c.e [dbType=postgresql, trace_id=7253053797189464882, txOriginName=ApplicationLifecycle:startServices:circlet.packages.server.db.PackagesMetaDBConnection:create locks table and check schema, dbKey=postgres] - Branch version from environment is null
[2023-07-07 12:33:04,221] ERROR [-pool] shared-3] circlet.platform.a.l.c.e [dbType=postgresql, trace_id=7253053797189464882, txOriginName=ApplicationLifecycle:startServices:circlet.packages.server.db.PackagesMetaDBConnection:create locks table and check schema, dbKey=postgres] - Deployment branch was not specified for production environment, env value is null
[2023-07-07 12:33:04,238] INFO [spatch thread 8] circlet.platform.a.l.r [dbType=postgresql, dbKey=postgres] - No migrations to apply, no changed tables, skip cluster lock
[2023-07-07 12:33:04,243] DEBUG [spatch thread 4] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.features.AppHealth
[2023-07-07 12:33:04,284] DEBUG [spatch thread 4] circlet.platform.a.a.f [] - Starting service: SupervisorClusterLockNotificationProvider
[2023-07-07 12:33:04,284] DEBUG [spatch thread 4] circlet.platform.a.a.f [] - Starting service: circlet.supervisor.server.db.SupervisorDBConnection
[2023-07-07 12:33:04,284] DEBUG [spatch thread 4] circlet.platform.a.a.f [] - Starting service: DebugLoggerUpdater
[2023-07-07 12:33:04,285] DEBUG [spatch thread 4] circlet.platform.a.a.f [] - Starting service: create default local S3 bucket
[2023-07-07 12:33:04,529] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.storage.PackagesDiskUsageSync
[2023-07-07 12:33:04,531] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.db.PackagesDBConnection
[2023-07-07 12:33:04,532] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.db.migration.PackagesDbMigrator
[2023-07-07 12:33:04,532] INFO [atcher-worker-2] .storage.PackagesDiskUsageSync [] - Sync package storage events in chunks of 2000 and send them not rare then 10000 ms
[2023-07-07 12:33:04,533] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: BackgroundLoggersDisabler
[2023-07-07 12:33:04,533] INFO [spatch thread 5] b.migration.PackagesDbMigrator [] - start migration process
[2023-07-07 12:33:04,534] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.platform.a.i.g
[2023-07-07 12:33:04,536] INFO [patch thread 10] circlet.platform.a.i.h [] - Own node: ClusterNode(id=99bde136aa3d, app=Packages, version=2023.2.0.128)
[2023-07-07 12:33:04,538] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: TracingNodeInfoHandler
[2023-07-07 12:33:04,539] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.platform.a.v.f
[2023-07-07 12:33:04,540] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.services.PackagesJobRegistry
[2023-07-07 12:33:04,542] INFO [patch thread 10] r.services.PackagesJobRegistry [] - Using redis bus id for packages job dispatcher: packages-common
[2023-07-07 12:33:04,559] INFO [patch thread 10] circlet.platform.a.m.u [] - registered dispatcher circlet.platform.a.m.OptimisticJobDispatcher@33eda2ee for crclt_packages_meta
[2023-07-07 12:33:04,564] INFO [spatch thread 4] m.a.m.AsyncExpirableSetWrapper [] - Acquired job-batches-packages-common expirable set
[2023-07-07 12:33:04,565] INFO [patch thread 10] circlet.platform.a.i.w [] - EventBusImpl: subscribe to packages-common
[2023-07-07 12:33:04,566] INFO [patch thread 10] circlet.platform.a.i.w [] - EventBusImpl: subscribed to packages-common
[2023-07-07 12:33:04,567] INFO [patch thread 12] circlet.platform.a.i.h [] - Node appeared: ClusterNode(id=99bde136aa3d, app=Packages, version=2023.2.0.128)
[2023-07-07 12:33:04,568] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.supervisor.server.services.migrations.SupervisorClusterDBMigrator
[2023-07-07 12:33:04,569] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.storage.PackagesDbAccessSync
[2023-07-07 12:33:04,571] DEBUG [patch thread 10] circlet.platform.a.a.f [] - Starting service: circlet.supervisor.server.db.DBTimeService
[2023-07-07 12:33:04,571] INFO [atcher-worker-2] r.storage.PackagesDbAccessSync [] - Sync package access events in chunks of 2000 and send them not rare then 10000 ms
[2023-07-07 12:33:04,579] DEBUG [spatch thread 8] circlet.platform.a.l.r [dbType=postgresql, trace_id=7253053797145810888, dbKey=postgres] - Create schema 'crclt_orgs' if not exists took 0.007s
[2023-07-07 12:33:04,580] INFO [spatch thread 8] circlet.platform.a.l.r [dbType=postgresql, dbKey=postgres] - Migrations which were disabled from supervisor are : []
[2023-07-07 12:33:04,594] WARN [-pool] shared-7] Exposed [dbType=postgresql, trace_id=7253053797028932608, txOriginName=ApplicationLifecycle:startServices:circlet.supervisor.server.db.DBTimeService:create locks table and check schema, dbKey=postgres] - The length of the binary column is not required.
[2023-07-07 12:33:04,594] WARN [-pool] shared-7] Exposed [dbType=postgresql, trace_id=7253053797028932608, txOriginName=ApplicationLifecycle:startServices:circlet.supervisor.server.db.DBTimeService:create locks table and check schema, dbKey=postgres] - The length of the binary column is not required.
[2023-07-07 12:33:04,596] DEBUG [spatch thread 1] rm.a.m.OptimisticJobDispatcher [] - Empty batch assigned, next wake up time: null, more due: false, has skipped: false
[2023-07-07 12:33:04,602] DEBUG [spatch thread 9] b.migration.PackagesDbMigrator [] - start migration process for org 1243897838375487330
[2023-07-07 12:33:04,603] DEBUG [spatch thread 9] b.migration.PackagesDbMigrator [] - found migrations [circlet.packages.maven.server.migrations.AddSnapshotInfoMigration@69b32816, circlet.packages.nuget.server.migrations.NuGetElasticSchemaMigration@19a5bd32, circlet.packages.npm.server.migrations.MigrateNpmPackageTagsFromElastic@7e635d30, circlet.packages.files.server.retention.FileLastAccessMigration@70ab1076, circlet.packages.container.server.migrations.ContainerTagsDbMigration@2875ed, circlet.packages.server.db.migration.PostgreOnlyIndexesMigration@345014f7].
[2023-07-07 12:33:04,611] INFO [spatch thread 6] circlet.platform.a.l.r [dbType=postgresql, dbKey=postgres] - No migrations to apply, no changed tables, skip cluster lock
[2023-07-07 12:33:04,620] DEBUG [spatch thread 2] circlet.platform.a.a.f [] - Starting service: circlet.packages.server.application.SqlPerformanceTelemetry
[2023-07-07 12:33:04,621] DEBUG [spatch thread 2] circlet.platform.a.a.f [] - Starting service: circlet.supervisor.server.services.orgs.LoadedOrgsServiceImpl
[2023-07-07 12:33:04,624] DEBUG [spatch thread 5] circlet.platform.a.i.m [] - begin cluster lock acquire with name db-migrate-packages-1243897838375487330-nuget
[2023-07-07 12:33:04,650] DEBUG [patch thread 10] b.migration.PackagesDbMigrator [] - cluster lock is acquired for packages-1243897838375487330-nuget
[2023-07-07 12:33:04,668] INFO [ main] Application [] - Start services took 0.646s
[2023-07-07 12:33:04,668] INFO [ main] Application [] - Application started in 6.741 seconds.
[2023-07-07 12:33:04,669] INFO [ main] pplication.PackagesApplication [] - Application started: io.ktor.server.application.Application@61c11339
[2023-07-07 12:33:04,683] DEBUG [patch thread 10] b.migration.PackagesDbMigrator [] - last applied migration for 1243897838375487330 / nuget is #0
[2023-07-07 12:33:04,690] INFO [patch thread 10] b.migration.PackagesDbMigrator [] - migration #11 AddSnapshotInfoMigration for 1243897838375487330 / nuget will be applied
[2023-07-07 12:33:04,691] DEBUG [patch thread 10] b.migration.PackagesDbMigrator [] - migration #11 AddSnapshotInfoMigration is applied for 1243897838375487330 / nuget
[2023-07-07 12:33:04,714] INFO [spatch thread 1] b.migration.PackagesDbMigrator [] - migration #14 NuGetElasticSchemaMigration for 1243897838375487330 / nuget will be applied
[2023-07-07 12:33:04,715] INFO [atcher-worker-1] Application [] - Responding at http://0.0.0.0:8390
[2023-07-07 12:33:04,715] INFO [atcher-worker-1] Application [] - Responding at http://0.0.0.0:9390
[2023-07-07 12:33:04,762] INFO [spatch thread 2] et.platform.a.o.b.ElasticIndex [] - Reindex task submitted. Task id: UkqEGvFqTfejLAhE_-oDGg:283
[2023-07-07 12:33:05,772] INFO [spatch thread 7] et.platform.a.o.b.ElasticIndex [] - Waiting for "Task UkqEGvFqTfejLAhE_-oDGg:283 completion", attempt 1
[2023-07-07 12:33:05,802] INFO [spatch thread 4] et.platform.a.o.b.ElasticIndex [] - "Task UkqEGvFqTfejLAhE_-oDGg:283 completion" finished
[2023-07-07 12:33:05,802] INFO [spatch thread 4] et.platform.a.o.b.ElasticIndex [] - Reindex task completed. Task id: UkqEGvFqTfejLAhE_-oDGg:283
[2023-07-07 12:33:05,802] DEBUG [spatch thread 4] b.migration.PackagesDbMigrator [] - migration #14 NuGetElasticSchemaMigration is applied for 1243897838375487330 / nuget
[2023-07-07 12:33:05,812] INFO [spatch thread 8] b.migration.PackagesDbMigrator [] - migration #17 MigrateNpmPackageTagsFromElastic for 1243897838375487330 / nuget will be applied
[2023-07-07 12:33:05,812] DEBUG [spatch thread 8] b.migration.PackagesDbMigrator [] - migration #17 MigrateNpmPackageTagsFromElastic is applied for 1243897838375487330 / nuget
[2023-07-07 12:33:05,824] INFO [patch thread 11] b.migration.PackagesDbMigrator [] - migration #18 FileLastAccessMigration for 1243897838375487330 / nuget will be applied
[2023-07-07 12:33:05,825] DEBUG [patch thread 11] b.migration.PackagesDbMigrator [] - migration #18 FileLastAccessMigration is applied for 1243897838375487330 / nuget
[2023-07-07 12:33:05,829] INFO [patch thread 12] b.migration.PackagesDbMigrator [] - migration #19 ContainerTagsDbMigration for 1243897838375487330 / nuget will be applied
[2023-07-07 12:33:05,829] DEBUG [patch thread 12] b.migration.PackagesDbMigrator [] - migration #19 ContainerTagsDbMigration is applied for 1243897838375487330 / nuget
[2023-07-07 12:33:05,837] INFO [spatch thread 3] b.migration.PackagesDbMigrator [] - migration #20 PostgreOnlyIndexesMigration for 1243897838375487330 / nuget will be applied
[2023-07-07 12:33:05,855] DEBUG [onPool-worker-2] circlet.platform.a.l.r [dbType=postgresql, trace_id=7253053801305072159, dbKey=postgres] - Create schema 'crclt_packages_1243897838375487330' if not exists took 0.005s
[2023-07-07 12:33:05,856] INFO [onPool-worker-2] circlet.platform.a.l.r [dbType=postgresql, dbKey=postgres] - Migrations which were disabled from supervisor are : []
[2023-07-07 12:33:05,863] INFO [onPool-worker-2] circlet.platform.a.l.r [dbType=postgresql, dbKey=postgres] - No migrations to apply, no changed tables, skip cluster lock
[2023-07-07 12:33:05,864] DEBUG [spatch thread 5] b.migration.PackagesDbMigrator [] - migration #20 PostgreOnlyIndexesMigration is applied for 1243897838375487330 / nuget
[2023-07-07 12:33:05,864] DEBUG [spatch thread 5] b.migration.PackagesDbMigrator [] - releasing cluster lock for packages-1243897838375487330-nuget
[2023-07-07 12:33:05,878] DEBUG [spatch thread 1] circlet.platform.a.i.m [] - cluster lock with name db-migrate-packages-1243897838375487330-nuget is released
[2023-07-07 12:33:05,879] DEBUG [spatch thread 7] b.migration.PackagesDbMigrator [] - success finish migration process for org 1243897838375487330
[2023-07-07 12:33:05,880] INFO [spatch thread 7] b.migration.PackagesDbMigrator [] - finish migration process
[2023-07-07 12:33:12,359] INFO [ redisson-3-7] circlet.platform.a.i.h [] - Node appeared: ClusterNode(id=298b14cecf69, app=Space, version=2023.2.0.128)
[2023-07-07 12:33:22,293] WARN [tty-8390-9390-1] pplication.PackagesApplication [requestId=1] - GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/pkgs?query=&size=50: 404
[2023-07-07 12:33:22,293] WARN [tty-8390-9390-2] pplication.PackagesApplication [requestId=0] - GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/stats: 404
[2023-07-07 12:35:03,610] WARN [tty-8390-9390-7] pplication.PackagesApplication [requestId=2] - GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/stats: 404
[2023-07-07 12:35:03,613] WARN [tty-8390-9390-1] pplication.PackagesApplication [requestId=3] - GET http://packages:9390/p/exampledev/~/api/repo/example-nuget/pkgs?query=&size=50: 404
Thank you! As a next step, please share the configuration settings in the `space.on-premises.conf` under `circlet.packages` section.
Also, checking Space server log for the following string would help understanding what is the current configuration:
```
2023-07-07 14:02:05.676 [main] INFO c.s.p.PackageRepositoryClientBase [] - Using Space Nuget Client: orgMode = SINGLE_ORG, packagesMode = CONTEXT, publicUrl = http://localhost:8390, internalUrl = http://localhost:9390
```
Again thank you.
Here is the circlet.packages in the space.on-premises.conf
server {
client_max_body_size 0;
server_name packages.examplesoft.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://10.10.0.25:8390/;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/packages.examplesoft.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/packages.examplesoft.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = packages.examplesoft.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name packages.examplesoft.com;
return 404; # managed by Certbot
}
Thanks! Could you please try to remove `internalUrl` from package types and leave only one `circlet.packages.internalUrl` property?
Like this:
```
packages {
internalUrl = "http://packages:9390"
notifications {
enabled = true
}
oauth {
clientId = "space-to-packages-client"
clientSecret = "mysecret="
}
types {
maven {
url = "https://packages.examplesoft.com"
}
nuget {
url = "https://packages.examplesoft.com"
}
npm {
url = "https://packages.examplesoft.com"
}
container {
url = "https://packages.examplesoft.com"
}
pypi {
url = "https://packages.examplesoft.com"
}
composer {
url = "https://packages.examplesoft.com"
}
dart {
url = "https://packages.examplesoft.com"
}
files {
url = "https://packages.examplesoft.com"
}
}
}
```
You are the man! Thank you so much this was driving me crazy! Again I really appreciate all the help!