Elastic unauthorized errors in Space on-premises k8s pod

Hello. We have a problem with error logs in k8s Space pods. 

Prerequisites:

  1. Fresh Space v.2023.2.0 installation in k8s (official chart was used).
  2. External Elastic v.8.9.1 is installed on dedicated host, HTTPS set up, certificate is issued by popular CA (trusted authority is  preinstalled to all modern OS), ApiKey is generated and used for access from Space (auth.apiKey parameter is set, other auth.* parameters - not).

When Space pods start, a lot of error messages about “missing authentication credentials” are written to logs (see below). Alongside with these error logs, Space successfully creates indexes in Elastic (i.e. ApiKey is working in other calls). 

[server dispatch thread 8] WARN  c.p.a.q.PlatformElasticClientImpl [dbType=postgresql, trace_id=7275764415987156920, orgDomain=space, orgId=5318541329097310114, dbKey=postgres] - Waiting 500 ms to retry
org.elasticsearch.client.ResponseException: method [GET], host [https://elastic.host:9200], URI [/_cat/indices/space-app*?h=index&format=json], status line [HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices/space-app*?h=index&format=json]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cat/indices/space-app*?h=index&format=json]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
    at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
    at org.elasticsearch.client.RestClient.access$1900(RestClient.java:108)
    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:397)
    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:393)
    at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87)
    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:833)

 

ApiKey is correct itself, I've tested manual request and it works

GET https://elastic.host:9200/_cat/indices/space-app*?h=index&format=json
Authorization: ApiKey <my key is here>
Accept: application/json

>>>

HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: application/json
Transfer-Encoding: chunked

[]
Response file saved.

Is there any problem in my config? It looks like Space ignores auth.* configuration in this method call only.

1
3 comments

same problem here. .

```[/_cat/indices/space-app*?h=index&format=json]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}```

0

I was forced to completely disable authentication on the elasitcsearch cluster in order to make it work 

0

Davidmirv, Dmitry Ivanoff  I apologize for the inconvenience caused. The new version with the fix is already on its way to production. Please stay tuned. 

1

Please sign in to leave a comment.