Saturday, September 26, 2020

Access Weblogic log information through Elasticsearch and Kibana

Elastic

Kibana


WebLogic Logging Exporter 

 https://github.com/oracle/weblogic-logging-exporter


Install WebLogic Logging Exporter 



Copy JAR to domain
cp ~/Downloads/weblogic-logging-exporter-1.0.0.jar /app/domains/base_domain/

Start Weblogic domain
[dave@dave ~]$ cd /app/domains/base_domain/
[dave@dave base_domain]$ ./startWebLogic.sh 

Access Weblogic console via browser at
http://localhost:7001/console/
Add StarUp class In the Administration Console, navigate to "Environment" then "Startup and Shutdown classes" in the main menu. Add a new Startup class. You may choose any descriptive name and the class name must be weblogic.logging.exporter.Startup. Target the startup class to each server that you want to export logs from.
base_domain]$ more config/config.xml

  <startup-class>
    <name>StartupClass-weblogic.logging.exporter</name>
    <target>AdminServer</target>
    <class-name>weblogic.logging.exporter.Startup</class-name>
  </startup-class>

Download org.yaml:snakeyaml https://search.maven.org/artifact/org.yaml/snakeyaml/1.27/bundle
 cp ~/Downloads/snakeyaml-1.27.jar /app/domains/base_domain/

Add CLASSPATH into bin/setDomainEnv.sh
ls snakeyaml-1.27.jar 
snakeyaml-1.27.jar
[dave@dave base_domain]$ grep snake bin/setDomainEnv.sh 
export CLASSPATH=$DOMAIN_HOME/weblogic-logging-exporter-1.0.0.jar:snakeyaml-1.27.jar:$CLASSPATH

Add WebLogic Logging Exporter config file
vi config/WebLogicLoggingExporter.yaml

more   config/WebLogicLoggingExporter.yaml
publishHost:  localhost
publishPort:  9200
domainUID:  base_domain
weblogicLoggingExporterEnabled: true
weblogicLoggingIndexName:  base_domain_wls
weblogicLoggingExporterSeverity:  Notice
weblogicLoggingExporterBulkSize: 1
weblogicLoggingExporterFilters:
- filterExpression:  'severity > Warning'

Install Docker ElasticSearch https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.2
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.2


ocker pull docker.elastic.co/elasticsearch/elasticsearch:7.9.2
7.9.2: Pulling from elasticsearch/elasticsearch
f1feca467797: Pull complete 
2b669da077a4: Pull complete 
e5b4c466fc6d: Pull complete 
3b660c013f1a: Pull complete 
0e7ad1133ad1: Pull complete 
b50d6e48f432: Pull complete 
bff3705905f9: Pull complete 
9509765886ad: Pull complete 
b7f06f509306: Pull complete 
Digest: sha256:2be3302537236874fdeca184c78a49aed17d5aca0f8fc3f6192a80e93e817cb4
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.9.2
docker.elastic.co/elasticsearch/elasticsearch:7.9.2

Run Docker Elastic


docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.9.2


Install Kibana https://www.elastic.co/guide/en/kibana/current/docker.html
docker pull docker.elastic.co/kibana/kibana:7.9.2
Start dev Kibana
docker run -d                           --name kibana                         -p 5601:5601                          --link elasticsearch:elasticsearch    -e "ELASTICSEARCH_URL=http://elasticsearch:9200"   docker.elastic.co/kibana/kibana:7.9.2

Docker containers
 docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
8ba52c43304f        docker.elastic.co/kibana/kibana:7.9.2                 "/usr/local/bin/dumb…"   4 seconds ago       Up 3 seconds        0.0.0.0:5601->5601/tcp                           kibana
9fdc569676e0        docker.elastic.co/elasticsearch/elasticsearch:7.9.2   "/tini -- /usr/local…"   6 minutes ago       Up 6 minutes        0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch

Weblogic start log when Elastic cannot be reached
======================= Weblogic Logging Exporter Startup class called
Reading configuration from file name: /app/domains/base_domain/config/WebLogicLoggingExporter.yaml
<Sep 26, 2020 10:24:49,146 AM CEST> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.> 
Config{weblogicLoggingIndexName='domain1-wls', publishHost='localhost', publishPort=9200, weblogicLoggingExporterSeverity='Notice', weblogicLoggingExporterBulkSize='1', enabled=true, weblogicLoggingExporterFilters=[FilterConfig{expression='null', servers=[]}], domainUID='domain1'}
javax.ws.rs.ProcessingException: java.net.ConnectException: Tried all: '2' addresses, but could not connect over HTTP to server: 'localhost', port: '9200'
 failed reasons:
  [0] address:'localhost/127.0.0.1',port:'9200' : java.net.ConnectException: Connection refused
  [1] address:'localhost/0:0:0:0:0:0:0:1',port:'9200' : java.net.ConnectException: Connection refused

    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:260)
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:254)
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:729)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:728)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:421)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:310)
    at weblogic.logging.exporter.LogExportHandler.executePutOrPostOnUrl(LogExportHandler.java:170)
    at weblogic.logging.exporter.LogExportHandler.createMappings(LogExportHandler.java:300)
    at weblogic.logging.exporter.LogExportHandler.<init>(LogExportHandler.java:56)
    at weblogic.logging.exporter.Startup.main(Startup.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeMain(ClassDeploymentManager.java:449)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeClass(ClassDeploymentManager.java:359)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager.access$100(ClassDeploymentManager.java:63)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager$1.run(ClassDeploymentManager.java:286)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager$1.run(ClassDeploymentManager.java:273)
    at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
    at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeClassDeployment(ClassDeploymentManager.java:272)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeClassDeployments(ClassDeploymentManager.java:253)
    at weblogic.management.deploy.classdeployment.ClassDeploymentManager.runStartupsAfterAppAdminState(ClassDeploymentManager.java:215)
    at weblogic.management.deploy.classdeployment.StartupClassPrelistenService.start(StartupClassPrelistenService.java:29)
    at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:76)
    at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
    at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:309)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
    at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:678)
    at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:54)
    at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
    at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:211)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
    at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:677)
    at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
    at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
    at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
    at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
    at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:651)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused by: java.net.ConnectException: Tried all: '2' addresses, but could not connect over HTTP to server: 'localhost', port: '9200'
 failed reasons:
  [0] address:'localhost/127.0.0.1',port:'9200' : java.net.ConnectException: Connection refused
  [1] address:'localhost/0:0:0:0:0:0:0:1',port:'9200' : java.net.ConnectException: Connection refused

    at weblogic.net.http.HttpClient.openServer(HttpClient.java:408)
    at weblogic.net.http.HttpClient.openServer(HttpClient.java:511)
    at weblogic.net.http.HttpClient.New(HttpClient.java:313)
    at weblogic.net.http.HttpClient.New(HttpClient.java:292)
    at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:295)
    at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:685)
    at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:42)
    at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1546)
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:366)
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:258)

Weblogic good start with Elastic running in Docker
======================= Weblogic Logging Exporter Startup class called
Reading configuration from file name: /app/domains/base_domain/config/WebLogicLoggingExporter.yaml
<Sep 26, 2020 10:35:23,780 AM CEST> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.> 
Config{weblogicLoggingIndexName='domain1-wls', publishHost='localhost', publishPort=9200, weblogicLoggingExporterSeverity='Notice', weblogicLoggingExporterBulkSize='1', enabled=true, weblogicLoggingExporterFilters=[FilterConfig{expression='null', servers=[]}], domainUID='domain1'}

dave@dave base_domain]$ curl "localhost:9200/_cat/indices?v"
health status index       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   domain1-wls SQqMMWxgTCi6nhWyDSmeyg   1   1         25            0     39.6kb         39.6kb


Verify that logs were posted


Running Elasticsearch and Kibana locally for testing



Kibana



Access Kibana at http://localhost:5601/ 


 Getting
Kibana server is not ready yet



Used newest version - but there are some issues
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Root mapping definition has unsupported parameters:  [doc : {properties={severity={type=keyword}, sequenceNumber={type=keyword}, subSystem={type=keyword}, level={type=keyword}, serverName={type=keyword}, messageID={type=keyword}, domainUID={type=keyword}, userId={type=keyword}, threadName={type=keyword}, machineName={type=keyword}, transactionId={type=keyword}, loggerName={type=keyword}, timestamp={type=date}}}]",
"at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:148) ~[elasticsearch-7.9.2.jar:7.9.2]",
"at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:136) ~[elasticsearch-7.9.2.jar:7.9.2]",
"at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:88) ~[elasticsearch-7.9.2.jar:7.9.2]",
"at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:421) ~[elasticsearch-7.9.2.jar:7.9.2]",
"... 23 more"] }

Older version 6.2.2
docker run -d                       --name elasticsearch              -p 9200:9200                      -p 9300:9300                      -e "discovery.type=single-node"   docker.elastic.co/elasticsearch/elasticsearch:6.2.2
d8308574a82fc0f36e43c71846db51c7e26e567c53927c207730cde61b2512f5




docker run -d                       --name elasticsearch              -p 9200:9200                      -p 9300:9300                      -e "discovery.type=single-node"   docker.elastic.co/elasticsearch/elasticsearch:6.2.2

docker run -d                           --name kibana                         -p 5601:5601                          --link elasticsearch:elasticsearch    -e "ELASTICSEARCH_URL=http://elasticsearch:9200"   docker.elastic.co/kibana/kibana:6.2.2
ba259c96db5564c02e80116820bb78db65fd2c8919e33449d3c0ed3a6da4fb5a

Running
 docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED              STATUS              PORTS                                            NAMES
ba259c96db55        docker.elastic.co/kibana/kibana:6.2.2                 "/bin/bash /usr/loca…"   34 seconds ago       Up 34 seconds       0.0.0.0:5601->5601/tcp                           kibana
d8308574a82f        docker.elastic.co/elasticsearch/elasticsearch:6.2.2   "/usr/local/bin/dock…"   About a minute ago   Up About a minute   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch

Weblogic log
======================= Weblogic Logging Exporter Startup class called
Reading configuration from file name: /app/domains/base_domain/config/WebLogicLoggingExporter.yaml
<Sep 26, 2020 10:53:58,858 AM CEST> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.> 
Config{weblogicLoggingIndexName='domain1-wls', publishHost='localhost', publishPort=9200, weblogicLoggingExporterSeverity='Notice', weblogicLoggingExporterBulkSize='1', enabled=true, weblogicLoggingExporterFilters=[FilterConfig{expression='null', servers=[]}], domainUID='domain1'}

Elasticsearch Kibana errors
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
    at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:182) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:813) ~[elasticsearch-6.2.2.jar:6.2.2]

[2020-09-26T09:03:43,980][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:140) ~[?:?]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_161]
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_161]
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_161]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_161]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_161]
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_161]
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_161]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_161]
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_161]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:141) ~[?:?]
    at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:123) ~[?:?]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:571) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:380) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:375) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:909) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockException(TransportReplicationAction.java:827) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:815) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:713) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:170) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:98) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:133) ~[?:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:350) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:462) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:175) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:514) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:103) [elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
    at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:182) ~[elasticsearch-6.2.2.jar:6.2.2]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:813) ~[elasticsearch-6.2.2.jar:6.2.2]

docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
<none>                                          <none>              233dc2c81588        14 hours ago        423MB
<none>                                          <none>              0aa94d9613a1        14 hours ago        615MB
oracle/serverjre                                8                   9e58399899c9        14 hours ago        359MB
docker.elastic.co/kibana/kibana                 7.9.2               ba296c26886a        3 days ago          1.18GB
docker.elastic.co/elasticsearch/elasticsearch   7.9.2               caa7a21ca06e        3 days ago          763MB
alpine                                          latest              a24bb4013296        3 months ago        5.57MB
oracle/weblogic                                 12.2.1.4-slim       9f08a7ac6c5a        11 months ago       817MB
<none>                                          <none>              fdf24d1eabc2        11 months ago       1.2GB
12214-weblogic-domain-generic                   latest              93dc22395c42        11 months ago       1.26GB
oracle/weblogic                                 12.2.1.4-generic    b67e91a22473        11 months ago       1.26GB
<none>                                          <none>              ba227fc7cf16        11 months ago       2.99GB
oraclelinux                                     7-slim              874477adb545        13 months ago       118MB
hello-world                                     latest              fce289e99eb9        21 months ago       1.84kB
docker.elastic.co/kibana/kibana                 6.2.2               87fb54648d86        2 years ago         985MB
docker.elastic.co/elasticsearch/elasticsearch   6.2.2               576825ff4f5a        2 years ago         574MB