Troubleshooting Connectivity Issues

This guide helps you diagnose and resolve connectivity problems between Arctic agents and clients.

Agent Not Responding

Symptoms

curl http://AGENT_IP:8080/livez times out or fails
CLI commands fail with "connection refused" or timeout errors

Diagnosis

1. Check Agent Service Status

systemctl status arctic-agent

If the service is not running:

systemctl start arctic-agent
journalctl -u arctic-agent -n 50

2. Check Agent is Listening

ss -tlnp | grep 8080

Expected output shows the agent listening:

LISTEN  0  4096  *:8080  *:*  users:(("arctic-agent",...))

3. Check Firewall Rules

# iptables
iptables -L INPUT -n | grep 8080

# nftables
nft list ruleset | grep 8080

# firewalld
firewall-cmd --list-ports

Ensure port 8080 is allowed.

Resolution

Start the agent service if stopped
Open port 8080 in the firewall
Check for conflicting services on port 8080

Peers Cannot Communicate

Symptoms

Peer handshake fails
Heartbeats not being received
Peers showing as unhealthy

Diagnosis

1. Test Direct Connectivity

From one agent host to another:

curl http://PEER_IP:8080/livez

2. Check UDP Tunnel Port

The IP tunnel (Tempest) uses UDP port 51840:

# Test UDP connectivity
nc -u PEER_IP 51840

3. Check Network Path

traceroute PEER_IP
mtr PEER_IP

Look for packet loss or high latency.

Resolution

Ensure both TCP 8080 and UDP 51840 are open between agents
Check for NAT issues if agents are on different networks
Verify routing between the networks

Traffic Not Being Routed

Symptoms

Services are created but traffic does not flow
Packets are not being captured by TProxy

Diagnosis

1. Check Service Configuration

arctic services list
arctic services get SERVICE_ID

Verify the service exists and has routes.

2. Check NFTables Rules

nft list ruleset | grep -A 10 arctic

Verify rules exist for your routes.

3. Check TProxy is Running

ps aux | grep pegasus
cat /opt/tillered/pegasus/config.json

4. Check IP Tunnel is Running

ps aux | grep tempest
cat /opt/tillered/tempest/config.yaml

Resolution

Trigger a config sync: arctic cluster sync
Verify routes match the traffic you expect to capture
Check that source/dest CIDRs are correct

MACVLAN Interface Issues

Symptoms

Service requires interface but none is created
Interface exists but has no IP address

Diagnosis

1. List Network Interfaces

ip link show
ip addr show

Look for interfaces named after service IDs (first 15 characters).

2. Check Network Manager Logs

journalctl -u arctic-agent | grep netmgr

Resolution

Verify the host has a suitable parent interface
Check the agent has CAP_NET_ADMIN capability
Ensure no interface name conflicts exist

DNS Resolution Issues

Symptoms

Cannot reach agents by hostname
DNS lookups fail within tunneled traffic

Diagnosis

nslookup HOSTNAME
dig HOSTNAME

Resolution

Verify DNS servers are configured correctly
Check if DNS traffic should be routed through Arctic
Add routes for DNS server IPs if needed

High Latency

Symptoms

Traffic through Arctic is slow
High ping times between services

Diagnosis

1. Measure Baseline Latency

# Direct connection
ping PEER_IP

# Through Arctic tunnel
ping DESTINATION_THROUGH_TUNNEL

2. Check for Packet Loss

mtr DESTINATION

3. Check Bandwidth Limits

arctic services get SERVICE_ID

Look for bandwidth_limit_mbps.

Resolution

Consider KCP transport for high-latency networks
Increase or remove bandwidth limits
Check for network congestion

Collecting Debug Information

When reporting issues, collect:

# Agent version
arctic version

# Agent status
systemctl status arctic-agent

# Agent logs
journalctl -u arctic-agent -n 100

# Network configuration
ip addr show
ip route show
nft list ruleset

# Arctic configuration
arctic peers list
arctic services list
cat /opt/tillered/pegasus/config.json
cat /opt/tillered/tempest/config.yaml

Connectivity Issues

On this page