deeparcher

deeparcher is a virtual machine running the Ctrl + All Application exposed to the Internet.

deeparcher is hosted by Hetzner (CAX11 with 4GiB Ram and 40GiB disk).

IP Addresses are:

Base Setup

Add keys to roots authorized_keys

Login to the machine via the Hetzner console as root (root password can be reset from the Hetzner management interface).

cat >> /root/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPo7jGtYCfcvevjRXRDETWxSpZKszAlDKHpygCAU6hh3 pukkamustard@posteo.net

Add additional SSH public keys as needed.

From now on you can connect via SSH.

Add users

Login via ssh as root and run following commands to create a new user with admin

# adduser pm
[...]
Adding user `pm' to group `users' ...

# usermod -G sudo pm

# mkdir -p /home/pm/.ssh/
# cat >> /home/pm/.ssh/authorized_keys
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPo7jGtYCfcvevjRXRDETWxSpZKszAlDKHpygCAU6hh3 pukkamustard@posteo.net

We have setup a user pm with sudo access (ability to run commands as root) that can login via ssh using the provided key.

Apt

Login as the newly created user (via ssh) and upgrade the sytem and install some nice tools to have lying around:

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install vim tmux mosh git rsync

Enable unattended updates:

sudo apt install unattended-upgrades

UFW

This sets up a basic firewall:

sudo apt install ufw

sudo ufw default deny
sudo ufw limit ssh
sudo ufw allow mosh
sudo ufw allow "WWW Full"

$ sudo ufw enable
Firewall is active and enabled on system startup

OpenSSH

For improved security we diable password authentication in SSH. Add the line

PasswordAuthentication no

to /etc/ssh/sshd_config.

Restart the SSH server with:

systemctl restart ssh

Ctrl+All application

We will run the ctrlall application as the user ctrlall:

sudo adduser ctrlall

Install the Python venv module:

sudo apt install python3-venv

Login as the user ctrlall with:

sudo su - ctrlall

The following steps need to be done as the user ctrlall.

Clone ctrlall repository

$ git clone https://codeberg.org/ctrlall/ctrlall.git

Create Python virtual environment and activate by default

$ python -m venv venv

This will create a Python virtual environment in the directory /home/ctrlall/venv.

To activate the environment run

. ~/venv/bin/activate

Now logout and re-login as the user ctrlall.

Install dependencies

pip install -r ctrlall/requirements.txt

You should now be able to run the ctrlall tool:

$ PYTHONPATH=~/ctrlall/src venv/bin/python -m ctrlall
usage: ctrlall [-h] [-d] [-v] [--database DATABASE]
               {import,export,ui,nips2rdf,bmf2rdf,stats,qb} ...

Ctrl + All: Computing the Social

options:
  -h, --help            show this help message and exit
  -d, --debug           Print debugging statements
  -v, --verbose         Print verbose information
  --database DATABASE   SQLite3 database for knowledge graph persistence (default:
                        `CTRLALL_DATABASE` environment variable or `ctrlall.db`)

commands:
  {import,export,ui,nips2rdf,bmf2rdf,stats,qb}
    import              Import data into the Knowledge Graph.
    export              Export RDF triples from Knowledge Graph
    ui                  Launc a web UI
    nips2rdf            Convert a NIPS data file to RDF
    bmf2rdf             Convert a HES Basic Master File to RDF
    stats               Print some Database statistics
    qb                  Inspect Data Cube (QB) data sets

Danger's over, Banana Breakfast is saved.

Note that we need to manually set the PYTHONPATH variable and use Python from the virtual environment.

We will additionally also install the gunicorn package for running the application:

pip install gunicorn

systemd user service

We will use systemd to manage user units that run the ctrall application and pull changes from git automatically.

First we need to make sure the ctrlall users systemd daemon is running even without a running session. As an admin user enabe lingering for the ctrlall user:

loginctl enable-linger ctrlall

When connecting via ssh (or for some related reason) the user session with some environment variables is not properly setup in the environment. Add following line to .bashrc to enable the XDG_RUNTIME_DIR even when connecting via ssh:

export XDG_RUNTIME_DIR="/run/user/$UID"

Reload the session of the ctrlall user (logout and login again) and then run:

$ systemctl --user status
● debian-ctrlall
    State: running
    Units: 83 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2024-03-27 13:50:06 UTC; 18h ago
  systemd: 252.22-1~deb12u1
   CGroup: /user.slice/user-1001.slice/user@1001.service
           ├─app.slice
           │ └─ctrlall.service
           │   ├─3410 /home/ctrlall/venv/bin/python3 /home/ctrlall/venv/bin/gunicorn -w 4 -b 127.0.0.1:5000 ctrlall.ui:app
           │   ├─3411 /home/ctrlall/venv/bin/python3 /home/ctrlall/venv/bin/gunicorn -w 4 -b 127.0.0.1:5000 ctrlall.ui:app
           │   ├─3412 /home/ctrlall/venv/bin/python3 /home/ctrlall/venv/bin/gunicorn -w 4 -b 127.0.0.1:5000 ctrlall.ui:app
           │   ├─3413 /home/ctrlall/venv/bin/python3 /home/ctrlall/venv/bin/gunicorn -w 4 -b 127.0.0.1:5000 ctrlall.ui:app
           │   └─3414 /home/ctrlall/venv/bin/python3 /home/ctrlall/venv/bin/gunicorn -w 4 -b 127.0.0.1:5000 ctrlall.ui:app
           └─init.scope
             ├─821 /lib/systemd/systemd --user
             └─822 "(sd-pam)"

If you don't get an error you're good to go.

ctrlall unit

This systemd service unit will start the ctrlall application.

Create the file /home/ctrlall/.config/systemd/user/ctrlall.service:

[Unit]
Description=Ctrl + All. User Interface

[Service]
Environment=PYTHONPATH=%h/ctrlall/src/
Environment=CTRLALL_DATABASE=%h/ctrlall.db
Environment=CTRLALL_ONLY_PUBLISHED_NOTES=True
ExecStart=%h/venv/bin/gunicorn -w 4 -b 127.0.0.1:5000 ctrlall.ui:app

[Install]
WantedBy=default.target

systemctl --user enable ctrlall
systemctl --user start ctrlall

ctrall-pull unit

This systemd service unit will pull most recent changes from codeberg.

Create the file /home/ctrlall/.config/systemd/user/ctrlall-pull.service:

[Unit]
Description=Pull changes from codeberg.org/ctrlall/ctrlall

[Service]
Type=oneshot
WorkingDirectory=%h/ctrlall
ExecStart=git pull
ExecStart=systemctl --user restart ctrlall

We also create a systemd timer unit that runs the pull regularly. Create the file /home/ctrlall/.config/systemd/user/ctrlall-pull.timer:

[Unit]
Description=Pull ctrlall changes regularly

[Timer]
OnBootSec=15min
OnUnitActiveSec=1h

[Install]
WantedBy=timers.target

Enable and start the timer:

systemctl --user enable ctrlall-pull.timer
systemctl --user start ctrlall-pull.timer

Making changes to the user service

When making changes to the systemd user service file you will need to run systemctl --user daemon-reload so that systemd picks up the changes.

Maging the ctrlall service

As user ctrlall.

To restart the ctrlall application:

systemctl --user restart ctrlall

To get logs:

journalctl --user -u ctrlall -r

To check status of the regular pull:

$ systemctl --user list-timers
NEXT                        LEFT       LAST                        PASSED UNIT               ACTIVATES           
Thu 2024-03-28 09:45:01 UTC 59min left Thu 2024-03-28 08:45:01 UTC 1s ago ctrlall-pull.timer ctrlall-pull.service

1 timers listed.

Nginx web server

As an admin user (not ctrlall):

sudo apt install nginx

/etc/nginx/sites-available/computingthesocial.net

server {
        listen 80;
        listen [::]:80;

        root /var/www/html;

        server_name computingthesocial.net deeparcher.computingthesocial.net;

        access_log off;

        location / {
                proxy_pass http://localhost:5000/;
        }
}

Certbot

Make sure DNS entries for computingthesocial.net and deeparcher.computingthesocial.net point to the VM. Then run:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d computingthesocial.net deeparcher.computingthesocial.net

This will get and install TLS certificates and setup a regular service to renew them.