Troubleshooting

Diagnostic commands 

Before starting diagnostics, run the basic set of checks:

# Общее состояние системы
citeck health

# Статус всех приложений
citeck status

# Логи конкретного приложения
citeck logs <app>

# Логи демона
citeck logs

# Логи демона через journald
journalctl -u citeck -f

# Состояние контейнеров Docker
docker ps -a --filter label=citeck.launcher=true

# Автоматическая диагностика с исправлением
citeck diagnose --fix

Note

Docker labels used by the launcher. All containers created by citeck are tagged with a set of labels for filtering and tracing:

citeck.launcher=true — indicates that the container is managed by the launcher (used with --filter)
citeck.launcher.workspace — workspace identifier
citeck.launcher.namespace — namespace identifier
citeck.launcher.app.name — logical application name
citeck.launcher.app.hash — deployment hash (used by reload to detect changes)
citeck.launcher.original-name — original name before normalization

Application stuck in STARTING status 

Symptoms: the application remains in the STARTING state for a long time and does not transition to RUNNING.

Causes and solutions:

Insufficient memory. Check available memory:
```
free -h
citeck describe <app>  # посмотрите memoryLimit
```
Increase heapSize via citeck setup resources or add more RAM.

Dependency not ready. The application may be waiting for another service:

citeck status  # проверьте статус зависимостей
citeck logs <dependency-app>

Configuration error. Check the application logs:
```
citeck logs <app> --tail 200
```
Restart the application:
```
citeck restart <app>
```

Keycloak fails to start 

Symptoms: Keycloak is in the START_FAILED or STARTING state.

Causes:

PostgreSQL unavailable. Keycloak depends on the database:

citeck status | grep postgres
citeck logs postgres --tail 50

Insufficient memory. Keycloak requires a significant amount of memory:

citeck describe keycloak  # проверьте heapSize
citeck setup resources    # увеличьте heapSize для keycloak

Corrupted database. If Keycloak reports migration errors:

citeck logs keycloak --tail 500 | grep -i "migration\|liquibase"

TLS certificate issues 

Self-signed certificate — browser warning

This is expected behavior. Add a browser exception to continue.

Let’s Encrypt cannot obtain a certificate

Check that ports 80 and 443 are reachable from the internet:
```
curl -v http://<your-host>
```
Check DNS records (if a domain name is used):
```
dig <your-domain>
```

Switch to a different TLS mode:

citeck setup tls
# Выберите "Self-signed HTTPS" или "Auto HTTPS"

Certificate expired

Let’s Encrypt certificates are renewed automatically. If renewal did not happen:

# Перезапуск демона обычно решает проблему
systemctl restart citeck

Forgotten administrator password 

The administrator password can be changed at any time:

citeck setup admin-password

The command applies the new password to all components (Keycloak, RabbitMQ, PgAdmin) without restarting containers.

Daemon not responding 

Symptoms: the citeck status and citeck start commands do not work.

Check the systemd service:
```
systemctl status citeck
```
Check the socket file:
```
ls -la /run/citeck/daemon.sock
```
Stale socket (socket exists but daemon is not running):
```
# Автоматическое исправление
citeck diagnose --fix

# Или вручную
rm /run/citeck/daemon.sock
citeck start
```
Note

citeck diagnose --fix only removes the stale socket but does not restart the daemon. After auto-fix, be sure to start the daemon again — systemctl start citeck (if the systemd service is installed) or citeck start -d (if not starting via systemd).

Daemon logs:

journalctl -u citeck --no-pager -n 100
# или
cat /opt/citeck/log/daemon.log

Restart via systemd:
```
systemctl restart citeck
```

Tip

When the systemd service is installed, prefer systemctl start citeck over citeck start -d. The former starts the daemon as a systemd unit and writes logs to journald, which simplifies debugging and ensures correct auto-start. citeck start -d remains useful when systemd is unavailable (e.g. in containers) or a manual start outside the service manager is needed.

Note

Daemon log locations:

File: /opt/citeck/log/daemon.log (with automatic rotation)
systemd/journald: journalctl -u citeck -f

Auto-recovery by the reconciler 

The daemon runs the reconciler, which continuously monitors the state of containers and automatically recovers failed applications:

Missing containers are recreated.
Failed liveness checks (3 consecutive misses) trigger a container restart.
Exponential backoff is used for restart retries: from 1 minute up to a maximum of 30 minutes.

Before manually restarting a problematic application, wait 1–2 minutes and check citeck status — the reconciler may bring the service back on its own. Manual citeck restart <app> makes sense only when the application does not recover automatically (e.g. the issue is in the configuration rather than a transient failure).

Applications detached via citeck stop <app> are not touched by the reconciler — they retain STOPPED status until manually started with citeck start <app>.

Insufficient disk space 

Symptoms: no space left on device errors, applications cannot start.

Check disk space:
```
df -h /opt/citeck
du -sh /opt/citeck/*
```

Clean up unused Docker resources:

# Показать, что будет удалено
citeck clean --volumes --images

# Выполнить очистку
citeck clean --force --volumes --images

Truncate a specific container’s logs (keeping the file but zeroing its contents):
```
# Найти путь к лог-файлу контейнера и обнулить его:
docker inspect --format='{{.LogPath}}' <container-name> | xargs sudo truncate -s 0
```
Alternatively, configure automatic log rotation globally — set logging driver limits in /etc/docker/daemon.json:
```
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}
```
After editing the file: systemctl restart docker. The settings apply to new containers; existing ones must be recreated (via citeck reload — the hash won’t change, but if you want to force recreation you can run citeck restart <app>).

Warning

Do not use docker system prune -f on a server running Citeck: the command will remove all unused containers, networks, and dangling images, which may affect infrastructure outside the launcher. For safe cleanup use citeck clean --force --volumes --images — it removes only orphaned launcher resources.
Daemon log rotation happens automatically. Check the size:
```
ls -lh /opt/citeck/log/
```

Port conflict 

Symptoms: the daemon or proxy cannot start, bind: address already in use error.

Check what is using the ports:
```
ss -tlnp | grep -E ':80|:443'
```

Stop the conflicting service:

# Например, если nginx занимает порт
systemctl stop nginx
systemctl disable nginx

If the port is occupied by a Docker container (e.g. a stray nginx, apache, or traefik container), find and remove it:

docker ps                   # найдите контейнер с нужным портом
docker rm -f <container>    # остановит и удалит контейнер

Or change the platform port:
```
citeck setup port
```

Daemon failure recovery 

Systemd automatically restarts the daemon on failure (Restart=on-failure). Docker containers continue running independently of the daemon.

After a restart, the new daemon instance picks up the running containers by comparing deployment hashes — application data and state are not lost.

If automatic restart did not help:

# Проверить логи
journalctl -u citeck --no-pager -n 200

# Ручной перезапуск
systemctl restart citeck

# Проверить, что контейнеры всё ещё работают
docker ps --filter label=citeck.launcher=true

Failed update rollback 

If the platform behaves incorrectly after a binary update:

citeck install --rollback

This command:

Restores the previous binary version from /usr/local/bin/citeck.bak
Restarts the daemon

Docker containers are not affected — they continue running with the previous images.

Health check in monitoring scripts 

For integration with monitoring systems (Zabbix, Nagios, Prometheus) use:

# Простая проверка (exit code: 0=OK, 8=unhealthy)
citeck health
echo $?

# JSON-вывод для парсинга
citeck health --format json

# Проверка конкретного приложения
citeck status --format json | jq '.apps[] | select(.name=="eapps") | .status'

Application in PULL_FAILED status 

Symptoms: the Docker image cannot be pulled.

Check internet connectivity:
```
curl -v https://registry.citeck.ru/v2/
```
Check registry credentials (for Enterprise):
```
docker login registry.citeck.ru
```
Retry the pull:
```
citeck restart <app>
```
For offline installation, make sure all images are loaded:
```
docker images | grep citeck
```

Contacting support 

To collect full diagnostics with a single command, run:

citeck dump-system-info

The command creates a citeck-dump-<timestamp>.zip archive in the current directory. The archive contains:

System information (uname, OS, memory, disk)
citeck version and platform status (status, health, diagnose)
Configuration: namespace.yml, daemon.yml, setup history
Daemon logs (last 5000 lines) + journalctl
Docker: ps -a, inspect of each citeck container, container logs (head 1000 + tail 2000 lines)
Any step execution errors — in errors.json

Send the resulting ZIP archive to the support service or attach it to an issue in the GitHub repository (usually < 5 MB).

Note

The --full option includes complete untruncated container logs. Use it only if the support team requests it (the archive may become significantly larger).

If dump-system-info is unavailable (older launcher version or the command does not work), collect diagnostics manually:

# Версия лончера
citeck version

# Состояние системы
citeck health --format json > health.json

# Статус приложений
citeck status --format json > status.json

# Логи демона
citeck logs --tail 5000 > daemon.log

# Логи проблемного приложения
citeck logs <app> --tail 5000 > app.log

# Информация о системе
uname -a > system.txt
docker version >> system.txt
free -h >> system.txt
df -h >> system.txt

Send the collected files to the support service or create an issue in the GitHub repository.

Insufficient memory (Enterprise)

The Enterprise bundle includes more than 20 Java applications and requires 24–32 GB of RAM. On a server with 16 GB the platform may be unstable: OOM Killer terminates applications and the server becomes unavailable.

Solutions:

Disable optional applications to save memory:

# Отключить приложения, которые не нужны (освобождает 4–5 ГБ)
citeck stop onlyoffice
citeck stop attorneys
citeck stop ai
citeck stop edi

Disabled applications will not start on reload. To re-enable them: citeck start <app>.

Increase the server RAM to 24–32 GB (recommended for Enterprise).
Reduce the heap of individual applications:
```
citeck setup resources
```