Collecting Installer State for Support Tickets¶

Installs and upgrades performed with the ivaap binary record per-step state under /opt/ivaap/.installer/. When a step fails or a deployment behaves unexpectedly after install, this state — combined with logs — is what SLB Support will request first.

Capture installer state with the dedicated subcommand:

ivaap status

This prints the current state, the last completed step, the failing step (if any), and the resume command. Include the full output in any support ticket.

The underlying state lives in:

/opt/ivaap/.installer/state.json

state.json is a plain JSON file written atomically under an exclusive lock. It contains step outcomes, inputs collected during install (FQDN, license server, selected datanodes), and rollback metadata. Attach it to support tickets alongside ivaap status output and any relevant component logs (see Log Files below).

For tickets involving a failed install or upgrade, include all of the following:

ivaap status output (terminal text)
/opt/ivaap/.installer/state.json
ivaap -v output (version, chart commit, package name)
Component log files for the impacted pods (see next section)

Log Files¶

If enabled, K3s single-instance IVAAP installations will write out all logs to files to

/opt/ivaap/ivaap-volumes/logs

in separate directories by component:

$ ls -1 /opt/ivaap/ivaap-volumes/logs/
activemq
backend
adminserver
postgres
proxy
scheduledtasks

Inside each directory there will be a live .log file, as well as rotated out files from previous history. INT recommends shipping these logs to your monitoring tool of choice for production deployments, either by scraping these files with something like Logstash/FileBeat or similar, or capturing them from the container outputs. When contacting SLB Support, please include the relevant log files for all components.

Capturing Versions¶

It’s often useful to check the versions of individual components, either for requesting support, comparing deltas before updating, or verifying against other documentation. Here are some various methods to find relevant versions/tags.

Helm Chart Version¶

Included in all IVAAP Helm Templates will be Chart.yaml. Starting with IVAAP 2026.1, the chart uses calendar versioning — version carries the combined IVAAP application and chart version, and the separate appVersion field has been dropped.

user@linux:/opt/ivaap/IVAAPHelmTemplate$ cat Chart.yaml
apiVersion: v2
name: ivaap
description: Helm chart for deploying IVAAP
type: application
version: 2026.1.0-1
maintainers:
  - name: Support
    email: customersupport.digital@slb.com
home: https://slb.com/

For additional build identifiers (full SDC package name and the Helm template commit baked into the binary), run ivaap -v:

$ ivaap -v
ivaap v2.0 (built 2026-06-08T17:09:39Z, commit c4e456a)
  IVAAP version:        2026.1.0-1
  Helm template commit: c4e456a
  Package name:         2026.1.0-1_IVAAPHelmTemplate_2026-06-07-476858

This combined output is what SLB Support will ask for first. However, sometimes updated builds may be provided piecemeal without being provided a full new helm template. If there are image tags in your deployment yaml that are overriding the default tags in the helm template, this information will be important to pass along to SLB support.

Image Tags¶

Image tags can be found in the Helm Template file values.yaml, and possibly within your configured deployment yaml file. For example you will see information for each node deployed as either a corenode or as a additional enabled datanode with the tag and reponame.

Example tags:

ivaapBackendNodes
  coreNodes:
    seednode:
      repoName: ivaap/backend/seednode
      tag: seednode-3.0-3-0af410b-20250620T150944Z
  dataNodes:
    dataimportnode:
      enabled: false
      repoName: ivaap/backend/dataimportnode
      tag: dataimportnode-3.0-3-0af410b-20250620T150944Z

A convinient script is provided in the helm template to collect a list of the images within the template values.yaml file:

user@linux:/opt/ivaap/IVAAPHelmTemplate$ ./collect-image-list.sh
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/activemq:activemq-6.1.7-96e1cba9-202506251620Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/adminserver:adminserver-3.0.42-569ce02-eclipse-temurin-21-jdk-alpine-202506262109Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/adminnode:adminnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/blobstoragenode:blobstoragenode-3.0-3-bb04d54-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/cloudstoragenode:cloudstoragenode-3.0-3-fd24b41-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/dataimportnode:dataimportnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/epsgnode:epsgnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesmasternode:geofilesmasternode-3.0-2-f38d7a2-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesreservoirsnode:geofilesreservoirsnode-3.0-2-f38d7a2-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesseismicnode:geofilesseismicnode-3.0-2-f38d7a2-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/messagingnode:messagingnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/mongonode:mongonode-3.0-6-77b1f74-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/mqgatewaynode:mqgatewaynode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/opensdublobstorager3node:opensdublobstorager3node-3.0-4-6d06752-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/opensducloudstoragenode:opensducloudstoragenode-3.0-4-6d06752-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/opensdus3r3node:opensdus3r3node-3.0-4-6d06752-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/playnode:playnode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/ppdmnode:ppdmnode-3.0-4-5fcc873-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/s3node:s3node-3.0-4-7b59464-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/seednode:seednode-3.0-3-0af410b-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/witsmlnode:witsmlnode-3.0-1-c2f557d-20250708T223606Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/admin-client:ivaap_admin-3.0.1
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:ivaap-dashboard-publish-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:ivaap-dashboard-publish-external-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:osdu-publish-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:ivaap-dashboard-standard-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:ivaap-dashboard-standard-external-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:osdu-standard-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/proxy:proxy-2024.0-cce98866-20250627T135324Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/task-release:witsmlliveupdatetask_release-3.0.1-1511246-202507091432-20250709T143406Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/task-release:witsmlpollingtask_release-3.0.1-1511246-202507091432-20250709T143406Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/task-release:witsmlwelltrajectorypollingtask_release-3.0.1-1511246-202507091432-20250709T143406Z

However, this will not tell the whole picture. A more accurate description of your deployment would be to use the icvt alias included with ivaap-helpers. icvt <namespace> command to help find all current versions that are deployed.

user@linux:/opt/ivaap/IVAAPHelmTemplate$ icvt ivaap
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/adminserver:adminserver-3.0.42-569ce02-eclipse-temurin-21-jdk-alpine-202506262058Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/activemq:activemq-6.1.7-96e1cba9-202506251620Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/admin-client:ivaap_admin-3.0.1
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/adminnode:adminnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/epsgnode:epsgnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/messagingnode:messagingnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/mqgatewaynode:mqgatewaynode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/playnode:playnode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/seednode:seednode-3.0-3-0af410b-20250620T150944Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesmasternode:geofilesmasternode-3.0-2-f38d7a2-20250626T003400Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesreservoirsnode:geofilesreservoirsnode-3.0-2-f38d7a2-20250626T003400Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/backend/geofilesseismicnode:geofilesseismicnode-3.0-2-f38d7a2-20250626T003400Z
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-standard:ivaap-dashboard-standard-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/frontend/dashboard-publish:ivaap-dashboard-publish-3.4.0
245634265005.dkr.ecr.us-west-2.amazonaws.com/ivaap/proxy:proxy-2024.0-cce98866-20250627T135324Z

Specific JAR Commit Hashes¶

The commit hash is also embedded in each individual IVAAP jars. This can be useful to trace specific issues for SLB developers.

To find the jar filename in a container within a backend deployment pod follow these steps to exec into it and grep for the right jar. Example:

[user@ivaap IVAAPHelmTemplate]$ kubectl get pods -n ivaap
NAME                                                  READY   STATUS    RESTARTS   AGE
adminserver-deployment-759fb78996-dmdld               1/1     Running   0          21d
ivaap-activemq-deployment-6b6956684f-9v5c7            1/1     Running   0          21d
ivaap-admin-deployment-57669dd46d-khvq7               1/1     Running   0          21d
ivaap-backend-deployment-6f5575b6ff-9wwnh             9/9     Running   0          21d
ivaap-dashboard-deployment-96b65978f-j6mgl            1/1     Running   0          21d
ivaap-dashboard-publish-deployment-6bc4d97ddf-pqjdb   1/1     Running   0          21d
ivaap-proxy-deployment-6875dcf849-9nbtq               1/1     Running   0          21d

[user@ivaap IVAAPHelmTemplate]$ kubectl get pod ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -o jsonpath="{.spec.containers[*].name}"
adminnode epsgnode messagingnode mqgatewaynode playnode seednode geofilesmasternode geofilesreservoirsnode geofilesseismicnode


[user@ivaap IVAAPHelmTemplate]$ kubectl exec -it ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -c messagingnode -- /bin/bash -c "ls /opt/ivaap/ivaap-playserver/deployment/ivaapnode/lib" | grep -i coredata
com.interactive.ivaap.IVAAPCoreData-3.0.1.jar

Then with the appropriate filename, you can get the commit hash of the jar:

$ unzip -q -c /opt/ivaap/ivaap-playserver/deployment/ivaapnode/lib/com.interactive.ivaap.IVAAPCoreData-3.0.1.jar META-INF/MANIFEST.MF | grep -i revision | cut -d '' -f 2
Git-Revision: 1511246

This also works for the scheduled tasks:

$ unzip -q -c /opt/ivaap/ivaap-playserver/scheduledtasks/ivaaptask/lib/IVAAPWitsmlAdapter-3.0.1.jar META-INF/MANIFEST.MF | grep -i revision | cut -d '' -f 2
Git-Revision: 4f3c49c

HTTP APIs¶

Finally, HTTP APIs also show version information for many components

For the Adminserver, IVAAPServer v2, the endpoint is: /IVAAPServer/api/v2/about

$ curl -s https://<DNS>/IVAAPServer/api/v2/about | jq .
{
  "name": "IVAAP Data Backend",
  "version": "3.0.42-569ce02",
  "majorVersion": "3",
  "minorVersion": "0",
  "buildVersion": "42-569ce02",
  "database": {
    "version": "PostgreSQL 15.10",
    "description": "PostgreSQL 15.10 (Ubuntu 15.10-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit"
  },
  "localTime": {
    "day": 5,
    "dayOfYear": 217,
    "month": 8,
    "year": 2025,
    "hour": 19,
    "minute": 25,
    "second": 15,
    "millisecond": 950
  },
  "localTimeZone": {
    "id": "GMT",
    "name": "(GMT0:00) GMT"
  },
  "api": {
    "versions": [
      {
        "id": "v2",
        "links": [
          {
            "rel": "OpenApi Specification",
            "name": "OpenApi Specifications",
            "relEntity": "v1/schema/int/openapispec",
            "children": false,
            "hasProjectEntityChildren": false,
            "isProjectEntity": false,
            "href": "/IVAAPServer/api/v2/openapispecs"
          },
...

For the backend: /ivaap/api/v1/cluster/services/versions

$ curl -s https://<DNS>/ivaap/api/v1/cluster/services/versions | jq .
{
  "timestamp": 1754422214432,
  "configuredServiceRoleCount": 7,
  "configuredServiceRoleNames": "admin,epsg,geofiles,geofiles::reservoirs,geofiles::seismic,messaging,mqgateway",
  "servicesVersionInfoCount": 7,
  "servicesVersionInfoValues": [
    {
      "name": "admin",
      "UUID": "990816d4-88f1-4a5a-9cdc-435647876ed2",
      "description": "Non-datasource specific node used for global cluster requests.",
      "versionMajor": "3",
      "versionMinor": "0",
      "versionBuild": "3-0af410b"
    },
    {
      "name": "epsg",
      "UUID": "819c378a-e654-491c-ab5f-1ecdc4ce1137",
      "versionMajor": "3",
      "versionMinor": "0",
      "versionBuild": "3-0af410b"
    },
    {
      "name": "geofiles",
      "UUID": "15b1bf3d-9eaa-4333-81c0-a80d1d4a1a3a",
      "description": "This node is packaged with native code: \\\\n IVAAPNativeLibs: f3e655e9 \\\\n IVAAPOpenSeismicNativeLibs: ",
      "versionMajor": "3",
      "versionMinor": "0",
      "versionBuild": "2-f38d7a2"
    },
...

Get requests to the above (or just entering them into the browser's URL bar) will return versions of running components

Backend Health API¶

Generally speaking, IVAAP deployment will be healthy if all pods are running and none of them are in a restart loop. For the backend nodes, ivaap-helpers includes an alias for checking the backend health API. The syntax is ibhealth <namespace>:

user@linux:/opt/ivaap/IVAAPHelmTemplate$ ibhealth ivaap
{
  "timestamp" : 1755618351429,
  "health" : "healthy",
  "available" : "admin,blobstorage,cloudstorage,dataimport,epsg,geofiles,geofiles::reservoirs,geofiles::seismic,messaging,mongo,mqgateway,ppdm,s3,witsml",
  "activeMQConnectionStatus" : {
    "healthy" : true
  }
}

Component-Specific Health Logic¶

Postgres

Check the testresults API for adminserver connection to the postgres database

/IVAAPServer/api/v2/testresults

{
  "from": {
    "actor": "akka://DefaultActorSystemControllerActorSystem:TestDataSourceByNameJsonActor:TestDataSourceByNameResponse for TestDataSourceByNameRequest{sourceType=team, sourceName=main}"
  },
  "success": true
}

Adminserver
- Port: 8080
- Internal Endpoint
  - <DNS>/IVAAPServer/api/v2/testresults
  - <DNS>/IVAAPServer/api/v2/servicetypes/trusts/services/main/testresults
  - <DNS>/IVAAPServer/api/v2/servicetypes/about/services/main/testresults
  - <DNS>/IVAAPServer/api/v2/servicetypes/datasourcescontrol/services/main/testresults
Reverse Proxy (NGINX)
- Ports: 80 (HTTP), 443 (HTTPS)
- Internal Endpoint
  - ivaap-backend-service:80
  - ivaap-adminserver-service:80
  - ivaap-dashboard-service:80
  - ivaap-dashboard-publish-service:80
  - ivaap-admin-service:80
ActiveMQ
- Ports: 61616 (TCP), 8161 (HTTP)
Seednode
- Port: 2550 (Default, envar based)
- Internal Endpoint
  - nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT
- Health Check: Seednode checks connectivity to its own cluster network port, making sure it is listening and ready for new connections (other nodes) to connect to it.
Playnode
- Port: 9000
- Dependencies: Depends on the healthy status of Seednode and ActiveMQ. For smooth operation, ActiveMQ should be initialized before the backend node cluster to avoid WebSocket connection issues.
- Internal Endpoint
  - nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT
  - ${IVAAP_COMMON_BACKEND_SERVER_HOST}api/v1/cluster/services/health
MQGatewaynode
- Dependencies: Like the playnode, MQGatewaynode also depends on the healthy status of Seednode and ActiveMQ. For smooth operation, ActiveMQ should be initialized before the backend node cluster to avoid WebSocket connection issues.
- Internal Endpoint
  - nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT
Other backend nodes (Adminnode, Epsgnode, Messagingnode, and all Data Connector Nodes)
- Dependencies: All depend on the healthy status of Seednode.
- Internal Endpoint
  - nc -z -w 5 $IVAAP_CLUSTER_HOSTNAME $IVAAP_CLUSTER_TCP_PORT

Manual Health APIs and Metrics¶

To verify the state of IVAAP, there are a few checks to be made to its components. These can be made in a browser, or the server’s shell. If none of these are working, even from the local shell, restart and check the proxy (NGINX) service’s logs, as all of these calls depend on it being operational. Some of these APIs are used in the container level health checks described in the Container Health Checks Chapter

#Restart the reverse proxy and check logs
kubectl delete pod ivaap-proxy-deployment-6875dcf849-9nbtq -n ivaap
kubectl logs ivaap-proxy-deployment-6875dcf849-9nbtq -n ivaap

Admin Server¶

The first Adminserver health check API can be found at /IVAAPServer/api/v2/about

/IVAAPServer/api/v2/servicetypes/datasourcescontrol/services/main/testresults

Note that unless these envars are set true, these APIs will require a valid bearer token:

IVAAP_DISABLE_SETUP_TEST_CREDENTIALS_CONTROL=true
IVAAP_DISABLE_SETUP_TEST_ENTITLEMENTS_CONTROL=true

The above environment variables are not safe for production. Please ensure to use these variables with caution and for troubleshooting purposes only!

Example (note ‘success:true’ at bottom)

$ curl -s https://<DNS>/IVAAPServer/api/v2/servicetypes/datasourcescontrol/services/main/testresults | jq 
{
  "from": {
    "actor": "pekko://DefaultActorSystemControllerActorSystem:TestHostedServiceByNameJsonActor:TestHostedServiceByNameResponse for TestHostedServiceByNameRequest{serviceType=datasourcescontrol, serviceName=main, sourceType=team}"
  },
  "success": true
}

Backend Admin Node¶

The first backend health check verifies cluster activity: https://<DNS>/ivaap/api/about

[user@ivaap IVAAPHelmTemplate]$ curl -s https://<DNS>/ivaap/api/about | jq
{
  "name": "IVAAP Data Backend",
  "version": "3.0.3-0af410b",
  "majorVersion": "3",
  "minorVersion": "0",
  "buildVersion": "3-0af410b",
  "localTime": {
    "day": 5,
    "dayOfYear": 217,
    "month": 8,
    "year": 2025,
    "hour": 20,
    "minute": 43,
    "second": 49,
    "millisecond": 366
  },
  "localTimeZone": {
    "id": "GMT",
    "name": "(GMT0:00) GMT"
  },
  "licenses": {
    "IVAAPServer": {
      "type": "expires",
      "state": "valid",
      "expiration": {
        "daysUntilExpiration": 20,
        "day": 25,
        "dayOfYear": 237,
        "month": 8,
        "year": 2025
      }
    }
  },
  "api": {
    "versions": [

This check shows the version of the admin node deployed. If this response comes back as "no available license found," check the license envar or contact SLB

Backend Data Nodes¶

After verifying the backend data node, you can check other parameters about the backend nodes with the following. This will return a health status, with available nodes as well as any nodes that are available but not configured.

$ curl -s https://<DNS>/ivaap/api/v1/cluster/services/health | jq
{
  "timestamp": 1650405721028,
  "health": "healthy",
  "available": "admin,dataimport,eia,epsg,geofiles,messaging,mongo,mqgateway,s3,witsml",
  "activeMQConnectionStatus": {
    "healthy": true
  }
}

As you can see in the example above, this API reports which nodes have joined and are active in the cluster, as well as the status of the ActiveMQ connection. A healthy ActiveMQ connection status is paramount in real-time updates for both projects and data.

Another useful request is https://<DNS>/ivaap/api/v1/cluster/services/metrics. This response includes detailed real-time performance metrics for each individual node:

[user@ivaap IVAAPHelmTemplate]$ curl -s https://<DNS>/ivaap/api/v1/cluster/services/metrics | jq
{
  "timestamp": 1754427603633,
  "configuredServiceRoleCount": 7,
  "configuredServiceRoleNames": "admin,epsg,geofiles,geofiles::reservoirs,geofiles::seismic,messaging,mqgateway",
  "servicesMetricsCount": 8,
  "servicesMetricsValues": [
    {
      "name": "admin",
      "UUID": "990816d4-88f1-4a5a-9cdc-435647876ed2",
      "startTime": "Mon Jul 14 20:32:15 GMT 2025",
      "startTimestamp": 1752525135057,
      "upTime": "22 days, 0 hours and 27 minutes",
When reaching out to INT support, capture versions and logs to help track down the issues.      "processCpuLoadPct": 0.13464991023339318,
      "processCpuTime": 17080230000000,
      "usedHeapKB": 318469,
      "committedHeapKB": 524288,
      "threadCount": 55,
      "daemonThreadCount": 12,
      "peakThreadCount": 62,
      "timestamp": 1754427603618
    },
...

Finally, the last check requires a valid authorization token to be included, so it is best to use the IVAAP Admin Client and verify the data sources are available at https://<DNS>/admin/#/ClusterActivities. This page serves as a front-end visualization of some of the preceding health checks.

Resetting passwords or reactivating locked accounts¶

Sometimes it may be necessary to directly manipulate the postgres database if all super admin accounts are locked. This can happen if too many incorrect login attempts are made. The adminserver container has baked-in functions in it's bashrc to assist with database manipulation. For example, running the command listSuperAdmins inside the adminserver pod will list all super admin users, including their active status:

user@linux:~$ kubectl exec -it adminserver-deployment-6b4bfb4497-c4bp6 -n ivaap  -- /bin/bash
adminserver-deployment-6b4bfb4497-c4bp6:/opt/ivaap/adminserver$ listSuperAdmins
                 guid                 | fname | lname |       email       | deleted | active |     username      |
--------------------------------------+-------+-------+-------------------+---------+--------+-------------------+
 415b1d1c-78ba-41b2-86a3-37067326156e | IVAAP | Root  | ivaaproot@int.com | f       | t      | ivaaproot@int.com |
(1 row)

If this active status is f for false, the command resetAllSuperAdminActive can be used to set the status back to true. This should allow the user to login as super admin again. Note that this does reset all super admin's status back to true.

These scripts are also availabe in the ivaap-helpers repo under scripts/ivaap-postgresdb-utils directory

Enable Additional Logging Levels (Debug)¶

At times, the default log levels are not sufficient to help diagnose issues. The following steps can be used to enable additional DEBUG logging in the different IVAAP components.

IVAAPData Backend Nodes and Tasks¶

The backend data nodes and tasks can adjust logging levels via the logback.xml configuration files in each node directory

[user@ivaap IVAAPHelmTemplate]$ kubectl exec -it ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -c adminnode -- /bin/bash
ivaap-backend-deployment-6f5575b6ff-9wwnh:/opt/ivaap/ivaap-playserver/deployment$ cd ivaapnode/conf/
ivaap-backend-deployment-6f5575b6ff-9wwnh:/opt/ivaap/ivaap-playserver/deployment/ivaapnode/conf$ vi logback.xml

For the tasks, they can be reached inside a container by:

[user@ivaap IVAAPHelmTemplate]$ kubectl exec -it ivaap-scheduledtask-deployment-5dc6494ff-fnlc5 -n ivaap -c witsmlpollingtask -- /bin/bash
/opt/ivaap/ivaap-playserver/scheduledtasks/ivaaptask $ cd conf
/opt/ivaap/ivaap-playserver/scheduledtasks/ivaaptask/conf $ vi logback.xml

In the xml file, the WARN values can be replaced with DEBUG, or lines for specific classes/wildcards to DEBUG and manipulate logs can be added very easily. No restart is required once changes are made to the file and saved; they will take effect immediately.

Capturing memory dumps (debugging)¶

Capturing java heap dumps can sometimes be useful to examine internal data allocations etc. INT may request you perform the steps below to output memory dumps to the relevant components’ logs directories. We recommend gzipping the resulting .hprof files before sending them to INT for analysis.

Before beginning, identify the container name for the target component you are wishing to create a jvm heap dump for. In the example below, the kubectl get pods -n ivaap command prints only the pod names within the ivaap namespace.cont Memory dumps can be captured from the “ivaap-backend-deployment”, “ivaap-scheduledtask-deployment”, and “adminserver-deployment” pods.

[user@ivaap IVAAPHelmTemplate]$ kubectl exec ivaap-backend-deployment-6f5575b6ff-9wwnh -n ivaap -c messagingnode -- bash -c 'pgrep java | xargs jmap -dump:format=b,file=/opt/ivaap/ivaap-playserver/deployment/logs/heap.hprof'
Dumping heap to /opt/ivaap/ivaap-playserver/deployment/logs/heap.hprof ...

This can then be copied out of the pod container onto your hosts file system.

kubectl cp ivaap/ivaap-backend-deployment-6f5575b6ff-9wwnh:/opt/ivaap/ivaap-playserver/deployment/logs/heap.hprof ./heap.hprof -c messagingnode

Common Issues with IVAAP Components¶

This section covers common pitfalls and bad states that can sometimes arise in various containerized environments.

502s in Browser¶

Sometimes, especially with older versions of our ingress proxy configuration, IVAAP components can become unavailable through the ingress proxy, despite being healthy internally. This can happen after a restart or update of individual components. In any case, often just restarting the proxy will resolve these issues:

kubectl delete pod ivaap-proxy-deployment-6875dcf849-9nbtq -n ivaap