monitoring process activity

load average is : 1 minute, 5 minutes, 15 minutes moving exponential average

1
2

user@host ~/workspace/cloudtv_/simulacrations > uptime
 17:12:56 up 80 days,  8:57,  4 users,  load average: 0.51, 0.38, 0.34

figure out how many logical CPUs you have, and divide the load average by that number, yield is percent of load over period of time. 100% is fully loaded, more is overloaded.

lscpu : see how many CPUs you have I have 4

controlling services and daemons

the six stages of systemd

units are initialized by systemd

  • service
  • socket
  • target
  • device
1
2
3
4

systemctl = see status of units
systemctl list-units --type=service = just see services
systemctl status $SERVICE_NAME = see status of a service
systemctl status sshd = check sshd status, disabled is default

masking and unmasking

using ssh

stores user fingerprints on user’s server in known_hosts, which is a record of servers visited

man in the middle, fake server configures itself with the ip address of servera, but fake server will have a different fingerprint, which will fail to match against the known-hosts file and warn about a man-in-the-middle

what may have happened instead is that servera was restored, and they forgot to back up the fingerprint information

you can add the correct known key to fix this error

can put settings in ~/.ssh/config to enforce strict host key checking and set the default auth method to ssh keys

configuring ssh key-based authentication

asymmetric cryptopgraphic mechanism

private key

  • for descrypting data
  • may be secured with passphrase public key
  • for encrypting data
  • publicly accessible
  • saved on target ssh server in home directory

connecting:

  • remote machine will generate a challenge, which is encrypted with your public key
  • if you can decrypt the data with your private key, you are allowed to make a connection
1
2
3
4
5

ssh-keygen # generate a key via prompts
ssh-copy-id -i ~/.ssh/custom_key.pub serverb # add your public key to the remote server; requests passphrase; adds public key to `~/.ssh/authorized_keys` on remote machine
ssh -i .ssh/custom_key serverb # connect with a custom key
ssh-agent # is used to temporarily store your private key for use
ssh-add ~/.ssh/custom_key # this no longer requires user to enter password every time

customizing openssh service configuration

locking down ssh daemon so it’s more secure; prevent the root user from logging in

1
2

ssh root@server # bad
vim /etc/ssh/sshd_config # NOT ssh_config, which is the global client config for ssh

change PermitRootLogin yes to PermitRootLogin no

systemctl reload ssh : reload the process

now logins will show permission denied

ultimately we want to prevent password auth by distributing a private key for users to auth against a target server, or having them provide their public keys

can also do:

ssh -o PubkeyAuthentication=no -o PasswordAuthentication=yes user@server

bonus

log bash history to a file

add to ~/.bashrc

1

export PROMPT_COMMAND='if [ "$(id -u)" -ne 0 ]; then echo "$(date "+%Y-%m-%d.%H:%M:%S") $(pwd) $(history 1)" >> ~/.logs/bash-history-$(date "+%Y-%m-%d").log; fi'

analyzing and storing logs

journal collects messages from:

  • output of booting
  • daemons start/run
  • syslog chatter

journal powered by systemd-journald, accessed by journalctl

1
2

user@host ~/workspace > systemctl status | grep journal
             └─systemd-journald.service

rsyslog takes syslog messages to /var/log

rsyslog protocol

  • facility
  • priority

authpriv.notice : can redirect messages /var/log/foo

rsyslog’s main config file is /etc/rsyslog.conf

/etc/rsyslog.d/ : can be used as a drop-in directory for configuration files

use logger -p local6.warn to send a message to local6 facility at warning priority

logrotate : prevents log files from growing too large

customize the policy at /etc/logrotate.conf and /etc/logrotate.d/

using the journal

1
2

journalctl -r # read journal in reverse
journalctl -u sshd.service # did not work on DO droplet, but worked on centos workpc

syslog priorities (least to most)

  • debug
  • info
  • notice
  • warn
  • err
  • crit
  • alert
  • emerg
1
2
3
4
5

journalctl -p err # show err and upward priority messages
journalctl -p err --since "2019-12-08 00:00:00" --until "2019-12-10 00:00:00" # midnight to midnight, dec 8 to 10
journalctl -o verbose # verbose output
journalctl _PID=1 # show processes for target PID, 1 is systemd
journalctl -b 1 # show me the data from the previous boot