苍穹elk日志量暴增问题原因排查

curl -H "Accept: application/json" -H "Content-type: application/json" -XPOST -d '{"query": {"match_all": {}},"aggs": {"per_count": {"terms": {"size":100,"field": "className.keyword"}}},"size": 0}'    http://ip:port/ierp-log-2024-06-05索引名/_search -u   ES用户:ES密码 （需要将es地址、索引名称修改为对应索引）

得到的json串

{"took":1611,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":142617214,"max_score":0.0,"hits":[]},"aggregations":{"per_count":{"doc_count_error_upper_bound":6142,"sum_other_doc_count":1428633,"buckets":[{"key":"kd.bos.db.BaseDB","doc_count":70079177},

{"key":"kd.fi.iep.task.ClearDayLogTask","doc_count":45750038},{"key":"kd.bos.orm.impl.ORMImpl","doc_count":15265243},{"key":"kd.bos.org.service.OrgServiceQueryImpl","doc_count":855633},{"key":"kd.bos.mvc.cache.PageCache","doc_count":686852},{"key":"kd.bos.dlock.curator.CuratorLockManager$1","doc_count":617193},{"key":"kd.bos.schedule.server.next.ScheduleNextMonitor","doc_count":580552},{"key":"kd.bos.schedule.message.JobProcessor","doc_count":563409},{"key":"kd.bos.schedule.notify.JobProcessorNotify","doc_count":479367},{"key":"kd.bos.schedule.message.mq.MQHelper","doc_count":427596},{"key":"kd.bos.mservice.form.FormServiceImpl","doc_count":411168},{"key":"kd.bos.license.service.LicenseServiceImpl","doc_count":275374},{"key":"kd.bos.servicehelper.operation.DeleteServiceHelper","doc_count":258050},{"key":"kd.bos.workflow.engine.impl.log.RunTimeLog","doc_count":221111},{"key":"kd.bos.schedule.executor.SchExecutorMessageCustomer","doc_count":213798},{"key":"kd.bos.schedule.message.ReadyTasks","doc_count":213798},{"key":"kd.bos.schedule.message.mq.MQMessageSender","doc_count":213798},{"key":"kd.bos.schedule.server.JobDispatcherProxy","doc_count":213442},{"key":"kd.bos.form.watermark.WaterMarkCache","doc_count":207939},{"key":"kd.bos.form.FormShowParameter","doc_count":204984},{"key":"kd.bos.schedule.server.ScheduleJobDispatchWorker","doc_count":196113},{"key":"kd.bos.workflow.engine.impl.persistence.deploy.DeploymentManager","doc_count":180522},{"key":"kd.bos.permission.cache.util.PermCommonUtil","doc_count":175171},{"key":"kd.bos.workflow.engine.impl.bpmn.listener.ListenerNotificationHelper","doc_count":166455},{"key":"kd.bos.mvc.cache.RootPageCache","doc_count":152800},{"key":"kd.bos.ksql.shell.KDStatement","doc_count":130277},{"key":"kd.bos.license.service.cache.LicenseCache","doc_count":111790},{"key":"kd.bos.web.actions.utils.FilePremissionUtil","doc_count":111002},{"key":"kd.bos.extplugin.internal.PluginDBProvider","doc_count":110703},{"key":"kd.bos.permission.cache.AdminUserCache","doc_count":107299},{"key":"kd.bos.form.SingleOrgContextHelper","doc_count":99836},{"key":"kd.bos.kdtx.common.exception.ExceptionLogger","doc_count":97451},{"key":"kd.tmc.fbp.service.inst.interest.AbstractIntCallStragety","doc_count":94508},{"key":"kd.bos.list.BillList","doc_count":73703},{"key":"kd.bos.workflow.engine.impl.asyncexecutor.schedule.WfJobHandler","doc_count":60791},{"key":"kd.bos.workflow.engine.impl.jobexecutor.JobUtil","doc_count":58601},{"key":"kd.bos.entity.cache.RedisModelCache","doc_count":57076},{"key":"kd.bos.mvc.FormConfigFactory","doc_count":53364},{"key":"xykj.api.SendOaToHandlePlugin","doc_count":52501},{"key":"kd.bos.workflow.engine.msg.quantitysum.impl.MessageQuantitySummaryServcieImpl","doc_count":46711},{"key":"kd.asc.cha.business.task.ThirdPayCheckTask","doc_count":46080},{"key":"kd.bos.message.service.handler.WeixinqyHandler","doc_count":45898},{"key":"kd.bos.openapi.base.dataservice.OpenApiDataServiceImpl","doc_count":44108},{"key":"kd.bos.workflow.design.util.DesignerModelUtil","doc_count":40750},{"key":"kd.asc.core.res.business.helper.ResourceRentStatusHelper","doc_count":33829},{"key":"kd.bos.facade.base.BaseFacade","doc_count":33428},{"key":"kd.bos.workflow.engine.impl.agenda.ContinueProcessOperation","doc_count":33192},{"key":"kd.bos.mvc.report.ReportCacheImpl","doc_count":32832},{"key":"kd.fi.bd.service.balance.VoucherQueryUtils","doc_count":28556},{"key":"kd.bos.permission.domainsplit.form.PermFormFacadeImpl","doc_count":27925},{"key":"kd.fi.er.business.utils.ErStdConfig","doc_count":27620},{"key":"kd.bos.workflow.taskcenter.plugin.ApprovalPluginNew","doc_count":25130},{"key":"kd.bos.mservice.qingshared.common.log.IERPLogger","doc_count":25124},{"key":"kd.tmc.cfm.common.service.calcint.LoanBillCalcNormalIntHandle","doc_count":25110},{"key":"kd.bos.workflow.util.concurrent.AbstractConcurrentInfoHandler$1","doc_count":24699},{"key":"kd.bos.entity.operate.OperateLog_Nothing","doc_count":24425},{"key":"kd.bos.workflow.engine.impl.concurrent.impl.ConcurrentDataServiceImpl","doc_count":24398},{"key":"kd.bos.portal.service.PortalMenuService","doc_count":23999},{"key":"kd.bos.util.SafeLogUtils","doc_count":23749},{"key":"kd.bos.workflow.design.graph.model.GraphModel","doc_count":23570},{"key":"kd.bos.schedule.next.observable.filter.ExecutorServerStatusFilter","doc_count":21979},{"key":"kd.bos.workflow.engine.impl.cmd.job.ExecuteAsyncJobCmd","doc_count":21905},{"key":"kd.bos.portal.pluginnew.QuickLaunchPlugin","doc_count":21584},{"key":"kd.bos.portal.pluginnew.common.MyAppAbstract","doc_count":21396},{"key":"kd.bos.permission.service.AbstractPermissionServiceImpl","doc_count":21349},{"key":"kd.bos.workflow.engine.impl.concurrent.ConcurrentDataThread","doc_count":21270},{"key":"kd.bos.workflow.engine.impl.cmd.task.ApprovalRecordOptimizationCmd","doc_count":21212},{"key":"kd.bos.workflow.engine.impl.util.FlowRecordUtil","doc_count":20278},{"key":"kd.bos.schedule.next.observable.filter.AbstractObservableDataFilter","doc_count":20131},{"key":"kd.fi.bcm.common.log.WatchLogger","doc_count":19764},{"key":"kd.bos.portal.util.PortalUsableFuncUtil","doc_count":19732},{"key":"kd.bos.workflow.facade.entity.WfAssignPersonsFacade","doc_count":19644},{"key":"kd.bos.bec.engine.cache.EvtCacheHelper","doc_count":19620},{"key":"kd.tmc.cfm.common.helper.CfmRateAdjustHelper","doc_count":19180},{"key":"kd.bos.bec.engine.consumer.BizEventConsumer","doc_count":18432},{"key":"kd.bos.permission.cache.BatchCheckPermCache","doc_count":18359},{"key":"kd.bos.workflow.engine.impl.el.JuelExpression","doc_count":17975},{"key":"kd.bos.entity.OpBizRuleSetReader","doc_count":17760},{"key":"kd.fi.dhc.formplugin.MyBillListPlugin","doc_count":17423},{"key":"kd.scm.sou.formplugin.task.SouInqueryAutoOpenTask","doc_count":17280},{"key":"kd.bos.workflow.engine.msg.handler.AbstractServiceHandler","doc_count":17032},{"key":"kd.bos.mvc.form.FormView","doc_count":16796},{"key":"kd.bos.org.controller.filter.OrgRelationFilter","doc_count":16648},{"key":"kd.bos.workflow.devops.statisticalanalysis.mq.WfDevopsConsumer","doc_count":16115},{"key":"kd.bos.message.utils.MessageSystemParamterUtils","doc_count":15807},{"key":"kd.bos.workflow.bpmn.converter.util.ModelConfigUtil","doc_count":15561},{"key":"kd.bos.portal.plugin.BizAppHomePlugin","doc_count":15472},{"key":"kd.bos.bec.engine.BecSystemParamterUtils","doc_count":15408},{"key":"kd.bos.mutex.impl.DataMutexImpl_Redis","doc_count":15344},{"key":"kd.bos.service.botp.facade.OperateBfFacade","doc_count":15228},{"key":"kd.bos.portal.util.OpenPageUtils","doc_count":15165},{"key":"kd.bos.workflow.engine.impl.interceptor.CommandInvoker","doc_count":15128},{"key":"kd.bos.workflow.engine.impl.asyncexecutor.schedule.ScheduleBasedJobManager","doc_count":15127},{"key":"kd.bos.bal.common.BalLogUtil","doc_count":14667},{"key":"kd.bos.orm.query.fulltext.QMatches","doc_count":14652},{"key":"kd.fi.bd.util.PerformanceWatch","doc_count":14083},{"key":"kd.bos.ais.util.DtsUtil","doc_count":13849},{"key":"kd.bos.permission.cache.UserMenuCache","doc_count":13749},{"key":"kd.bos.ca.SignService","doc_count":13555},{"key":"kd.bos.workflow.engine.impl.el.ExpBooleanOperationImpl","doc_count":13304}]}}}

可以看到，kd.bos.db.BaseDB、kd.fi.iep.task.ClearDayLogTask执行次数最多，达到千万次级别

查看是否是系统调度任务频繁执行导致日志增长

首先来到调度作业列表

点击新增，

选择执行程序，输入 kd.fi.iep.task.ClearDayLogTask 在列表中查询，得到任务编码

通过编码查询调度作业

查看调度作业计划

查看调度作业计划执行计划

ClearDayLogTask该任务，每隔2天执行一次，每次在凌晨3点。但是日志中6月5号还在打印日志，可能清理出现问题，通过查询同一条traceID在6月4号和5号 是否均有执行记录，判断执行出错。通过查找ERROR级别日志，找到报错信息

通过删除智能核算日志数据出错关键字筛选日志，发现报错很多

ClearDayLogTask-系统于[2024-06-04 23:57:59]删除智能核算日志数据出错，原因：kd.bos.exception.KDException: CAUSE: kd.bos.orm.query.EntityNotExistsException: Entity iep_intellexecdetaillog get error: number为iep_intellexecdetaillog的元数据不存在。 appgroup:defaultGroup

at kd.bos.orm.impl.ORMConfiguration.doGetDataEntityType(ORMConfiguration.java:296)

at kd.bos.orm.impl.ORMConfiguration.getDataEntityType(ORMConfiguration.java:255)

at kd.bos.orm.impl.ORMConfiguration.innerGetDataEntityType(ORMConfiguration.java:245)

at kd.bos.orm.impl.ORMImplStandard.doCreateMultiQuery(ORMImplStandard.java:403)

at kd.bos.orm.impl.ORMIm...

查看对应的数据中心ID，发现是多数据中心中 的2个数据中心，未升级财务云6.0.7，导致元数据不存在，定时任务一直执行失败，然后不断重试。