Operator guide

How to read this guide

• Every command is run on the server as root (sudo -i).
• aishie.example.edu stands for your server’s name. The school in the examples, 示範大學 (“Example University”), its AD FS (adfs.example.edu), its learning management system (lms.example.edu) and its bucket (aishie-files) are all made up.
• Messages in code blocks are the commands’ own output. Blocks marked as examples are put together from that output; the real content (versions, digests, times) will differ.

Whatever the site marks “Set by the server’s operator” is set here. What the site tells administrators does not name the settings: hovering over the ⓘ beside it shows “For the server’s operator: …” with the setting, which an administrator can pass on to you.

Platform administrators and department administrators, who work in the site’s Administration pages, should read the Administrator guide: sign-in, AI and documents, People & agents, Export conversations and the rest are explained there. Students, teachers and agent owners should read the User guide.

Versions these guides describe

The three guides (the User guide, the Administrator guide and this one) are written for AIshie as of 10 October 2026: the latest release, 0.2.0, and the features merged into each repository’s main since, which are not released yet (the second table below). The new and retaken screenshots are of these versions on main; the others are of 0.2.0, whose screens have not changed.

The latest release, 0.2.0, was published for all three images together on 3 October 2026:

0.2.0 adds requesting changes, the course materials an answer relied on, the whole class’s gradebook, the PDF viewer on a phone and back closing what covers the page, and changes how the site looks and some of its wording: single sign-on comes first on the sign-in page, a course’s tabs sit in one row, the account menu has About AIshie, and the chat window says who reads a conversation. It brings Core’s migrations 0028 and 0029 and the runtime’s migration 0015, none of which needs anything of you; update Core first, then the runtime, then the front end (see Upgrading to 0.2.0).

The versions merged into main after 0.2.0, which servers following edge already run; servers following stable wait for the next release:

They add group assignments and peer evaluation, deleting an assignment for good, upstream routing for OpenRouter models on the school’s plan, and what an answer does when it runs out of budget, with notices in the asker’s language; AIShie-Deploy can put the server behind Cloudflare. In the databases, they bring Core’s migrations 0030, 0031 and 0032 and the runtime’s migration 0016, none of which needs anything of you (see Core’s migrations 0030, 0031 and 0032). edge updates in the usual order: Core, then the runtime, then the front end.

The release before it was the first, 0.1.0, published for all three images on 2 October 2026; the feature list was first written for it:

AIShie-Deploy has no releases: the table gives its main on that day, which this guide’s steps on the server are written for.

A server that follows edge may run newer versions than these. The version and commit your server runs are in Core’s and the runtime’s /healthz, and in the front end’s /version.json (see Health checks); on the site, About AIshie in the account menu shows them too.

Contents

1. Architecture at a glance: the five containers, routing, where things are
2. Installing a new server: before you start, setup-server.sh, the first administrator, behind Cloudflare
3. Settings files: aishie.env, core.env, runtime.env, postgres.env
4. Where uploaded files are kept: disk or an S3 bucket, CORS, moving files
5. The agent runtime: its credential, administrators, ceilings, each answer’s budget, checks
6. Model providers and keys: the school AI plan
7. Single sign-on (server level): OIDC_*, SECRETS_KEY, private addresses
8. Letting other sites embed AIshie
9. Updates: edge and stable: aishie-update, upgrading to 0.2.0
10. Database migrations and rolling back: including Core’s migrations 0027 to 0032, and the runtime’s migrations 0015 and 0016
11. Backups and restoring
12. Rotating secrets
13. Logs and monitoring
14. Troubleshooting
15. Appendix: commands and ports

The five containers

Caddy routes requests by path:

• /v1/*, /mcp, /mcp/*, /healthz, and the OAuth and OpenID discovery paths under /.well-known/: to Core. These responses are not compressed, because MCP streams.
• /runtime/api/*: to the agent runtime’s API (9091), with the browser’s Cookie header removed first. This API accepts only the assertions Core issues for a signed-in person, never the sign-in cookie. The front end’s AI and documents page and the dialogs for hosting an agent on AIshie reach the runtime this way.
• Everything else: to the front end (web).

The runtime’s port 9090 (health checks, metrics and /status) is never exposed through Caddy.

The agent runtime (the site in English calls it the agent service) hosts the site’s agents, converts Office files to PDF, reads scanned documents and, once the administrators turn it on, transcribes files into text versions (see The agent runtime).

The network and HOST

• The stack’s network has its own subnet, AISHIE_SUBNET (172.30.83.0/24), with Caddy fixed at AISHIE_CADDY_IP (172.30.83.10). This is the only proxy Core and the runtime’s API trust: they take a user’s address only from the X-Forwarded-For that Caddy sends. Caddy writes a single address there: the connection’s own, or, behind Cloudflare, the visitor’s (see Behind Cloudflare).
• Caddy has one more alias on the network: the server’s name, HOST. When the runtime calls https://aishie.example.edu, it reaches Caddy directly and gets the same certificate a browser does, without going out to the server’s public address and back in (which some providers do not route).
• If the subnet clashes with another network on the server, change both settings in /etc/aishie/aishie.env, then run aishie compose down && aishie compose up -d. The images and the data stay as they are.
• The server pulls its own updates: aishie-update checks every 5 minutes (see Updates). GitHub’s workflows never connect to the server.

Where things are

Before you start

• A server: a fresh Ubuntu 24.04 or later, with 2 CPUs, 4 GB of memory and 40 GB of disk to start. The server will hold grades and students’ work, so choose a provider and region your school allows.
• A firewall: the provider’s firewall must allow 80/tcp, 443/tcp and 443/udp. If ufw is on on the server, setup-server.sh opens these itself; ports that Docker publishes are not limited by ufw anyway.
• A DNS name: a name that points at the server (an A record; add AAAA if the server has IPv6). As soon as the name resolves to the server, Caddy gets its certificate.
• Docker: no need to install it first. setup-server.sh installs Docker Engine and the compose plugin (compose 2.24 or later is needed): on Ubuntu 24.04 it uses Ubuntu’s docker.io and docker-compose-v2; when Ubuntu’s packages are too old, or Docker’s docker-ce is already installed, it uses Docker’s apt repository instead. It says which one it used.
• Accounts: none needed. The AIShie-Deploy repository and the three images are public: the server reads and pulls them with no key, token or login.

Decide four things first

1. edge or stable (see The two channels): edge follows the newest version of each repository’s main that has passed its checks (:edge), and updates itself; stable follows only the releases (X.Y.Z) you set by hand. Before you choose stable, make sure all three images have published the release you want to use. The first release, 0.1.0, came out on 2 October 2026 for all three images, each beside the same Core: on stable, use 0.1.0 or a later release for X.Y.Z below.
2. Where uploaded files go (see Where uploaded files are kept): this server’s disk, or an S3-style bucket. Once this is set, files can only be moved with aishie storage migrate.
3. Single sign-on (see Single sign-on): none; the operator’s provider, set in core.env; and/or providers that platform administrators add on the site.
4. Keys for the models (see Model providers and keys): only agent owners’ own keys, or a school AI plan as well.

Running setup-server.sh

1. As root, in root’s home directory, get AIShie-Deploy:

   git clone https://github.com/AIShie-Education/AIShie-Deploy.git

2. Run the setup with the server’s DNS name and the environment (edge or stable):

   sh AIShie-Deploy/setup-server.sh aishie.example.edu edge

   When someone is at the terminal, it first asks where uploaded files go: pressing Enter keeps them on this server’s disk; you can also choose Amazon S3, Cloudflare R2, Backblaze B2 or another S3-compatible service, then enter the bucket and keys. When nobody is there to answer, give options instead (see Setting up a bucket).
3. Wait for it to finish. At the end it lists what is left to do (see the example below).

It does these things in order; in its output, each step is marked by a heading that starts with ==:

The secrets it generates: the passwords of the two database roles and of the PostgreSQL superuser (24 random bytes each, hex), SIGNING_KEY (32 bytes, hex), SECRETS_KEY (32 bytes, base64) and kek/v1 (32 bytes, base64). They are written only to files under /etc/aishie and never printed.

Example: what setup-server.sh prints at the end on a new edge server that keeps files on disk:

== Done. What is left
1. Point aishie.example.edu at this server in DNS (A, and AAAA if it has IPv6). Caddy gets
   its certificate once the name resolves here:
     curl https://aishie.example.edu/healthz
2. The first administrator, once Core runs (aishie-update --status): it asks
   for a name, an email and a password, then restarts Core, and the
   administrator signs in at https://aishie.example.edu with that email and password:
     aishie admin
3. Agents (README.md, Hosting agents): each is hosted one way for good, chosen
   when it is made. A runtime agent's owner has this server's runtime host it
   from the site, and the runtime is issued its token by its id: nobody pastes
   one. An operator's own runtime agent is a YAML in /etc/aishie/runtime/agents,
   readable by group 65532, that names its agent_id and holds no token:
     install -g 65532 -m 640 tutor.yaml /etc/aishie/runtime/agents/
     aishie runtime check --live && aishie compose kill -s HUP runtime
4. Keep a copy of /etc/aishie somewhere else, encrypted, and apart from the
   database's backups: it holds SIGNING_KEY, SECRETS_KEY and the runtime's key
   (kek/v1), which no backup of the database can bring back, and the
   runtime's credential for Core (README.md, What to keep off the server).

Updates come by themselves from now on: aishie-update --status says what runs
and how the last check went; journalctl -u aishie-update has every run.

If you chose stable

Stable’s three channels start out empty, so the first update does not run. setup-server.sh lists this step at the end: in /etc/aishie/aishie.env, set CORE_IMAGE, RUNTIME_IMAGE and WEB_IMAGE to a release, run aishie-update, then run aishie runtime-credential to issue the runtime its credential.

CORE_IMAGE=ghcr.io/aishie-education/aishie-core:X.Y.Z
RUNTIME_IMAGE=ghcr.io/aishie-education/aishie-agent-runtime:X.Y.Z
WEB_IMAGE=ghcr.io/aishie-education/aishie-frontend:X.Y.Z

Replace X.Y.Z with the version you want to use.

DNS and the first administrator

1. Point the server’s name at the server (an A record, plus AAAA if it has IPv6). As soon as the name resolves to the server, Caddy gets its certificate. Check that Core answers:

   curl https://aishie.example.edu/healthz

   It answers with JSON that includes status, version and commit.
2. Create the first administrator, the account you will sign in to the site with:

   aishie admin

   It asks for a name, an email and a password (entered twice and not shown; at least 10 characters), creates the Root account and the system account, then restarts Core so that its background jobs start as the system account.
3. Sign in at https://aishie.example.edu with that email and password.

Example (excerpt):

Name: 陳大文
Email (to sign in with): admin@example.edu
Password (10 characters or more):
The password again:
aishie: restarting Core, so that its background jobs start
aishie: done. Sign in at https://aishie.example.edu with admin@example.edu and that password.

• People have no API tokens, administrators included: only agents do.
• From a script: give the password on standard input to aishie core bootstrap --name … --email … --password-stdin >/dev/null, then run aishie compose restart core. The password never appears on the command line.
• What comes next (terms, departments, courses, sign-in methods) is done by a platform administrator on the site: point them to first-time setup in the Administrator guide.

After setup

1. Encrypt a copy of /etc/aishie and keep it off the server, apart from the database backups (see What to keep off the server).
2. The next day, check that the nightly backup ran:

   ls -l /var/backups/aishie/*-daily-*

Updating AIShie-Deploy’s own files

The images update themselves, but the stack’s files (compose.yaml, aishie-update, aishie, the systemd units) do not. To install a newer version:

git -C AIShie-Deploy pull
sh AIShie-Deploy/setup-server.sh aishie.example.edu edge

It installs the new files and leaves the settings, secrets and data as they are. It adds something in only two cases: when core.env has no SECRETS_KEY, it adds one; when the runtime’s credential file is missing or empty, it issues a new one. It says so when it adds anything; copy /etc/aishie again when it does.

Changing the server’s name

1. Change HOST in /etc/aishie/aishie.env, then run aishie compose up -d: Core, the runtime and Caddy are recreated with the new name, and Caddy gets a certificate for it.
2. With a bucket, its CORS rule names the old address: change it to the new one (see The CORS rule).
3. The redirect URI registered at each single sign-on provider is the old address too: change it at the provider to https://<new name>/v1/auth/sso/callback.

Behind Cloudflare (for visitors in mainland China)

A server’s own IP address can be blocked in mainland China, and then nobody there reaches the site, whatever its name. A server in Singapore, for example, could not be reached at all from China Telecom, China Unicom or China Mobile, while Hong Kong, Singapore, Japan and the United States reached it. With the server’s name proxied by Cloudflare (the orange cloud in its dashboard), visitors reach Cloudflare’s addresses, and Cloudflare reaches the server.

On the server

1. Update AIShie-Deploy’s own files first (see After setup): git -C AIShie-Deploy pull, then setup-server.sh. Without FRONT_PROXY, this changes nothing visitors see.
2. In /etc/aishie/aishie.env, add:

   FRONT_PROXY=cloudflare

3. Run (running setup-server.sh again does the same):

   aishie front-proxy

From then on:

• Caddy believes Cloudflare’s addresses alone about who the visitor is: on a connection from one of them, the visitor is the address in CF-Connecting-IP, which Cloudflare sets on every request. A connection from anywhere else is its own client, whatever headers it brings.
• Core and the runtime are given the visitor’s address, not Cloudflare’s: Core counts sign-in attempts by it, and the runtime’s API records and limits by it.
• Cloudflare’s addresses are fetched from https://www.cloudflare.com/ips-v4 and /ips-v6 each time aishie front-proxy runs, and every week by aishie-front-proxy.timer. A list is taken only whole; when a fetch fails or brings anything else, the last list fetched stays in force (/var/lib/aishie/cloudflare-ips), or, on a server that never fetched one, the list that comes with AIShie-Deploy, and it says which. Caddy is reloaded only when the list or a setting changed; if Caddy refuses the new settings, the files are put back, and Caddy goes on as before.
• What aishie front-proxy writes is two files in /opt/aishie/caddy/front-proxy/: to change the settings, edit aishie.env, not them. Last, it asks http://HOST/.well-known/acme-challenge/, as Let’s Encrypt does, and says what answered (see The certificate, below).

Optionally, once the record is proxied by Cloudflare, add:

FRONT_PROXY_ONLY=yes

and run aishie front-proxy again: a request that comes neither from Cloudflare nor from a private address (the stack’s own network, and Docker’s proxy) is refused with 403, so that nobody reaches the site past Cloudflare. Set before the record is proxied, it refuses everyone who reaches the server directly, and aishie front-proxy warns you. It does not hide the server: the server’s address still answers (with the refusal).

In Cloudflare’s dashboard

By someone who may change the domain’s settings:

1. SSL/TLS › Overview: Full (strict). Not Flexible: Caddy redirects HTTP to HTTPS, and Flexible would loop for ever. The mode is the whole domain’s: every other proxied name in it that has a server of its own must present a valid certificate too.
2. DNS › Records: the server’s name as an A record to the server’s IPv4 address, Proxied, and no AAAA record to the server. Visitors still reach Cloudflare over IPv6.
3. Let plain HTTP requests to /.well-known/acme-challenge/ reach the server, so that it can renew its certificate: SSL/TLS › Edge Certificates › Always Use HTTPS: Off. Caddy redirects HTTP to HTTPS itself, so visitors still end up on HTTPS. If other names in the domain need it on, turn it off all the same and add, under Rules › Redirect Rules, one that does the same but for this path (AIShie-Deploy’s README, “Behind Cloudflare”, gives its full condition).
4. Leave these as they are, or check them: no rule that caches everything (if there ever is one, it must leave out /v1/*, /mcp*, /runtime/api/* and /healthz); Under Attack mode, Bot Fight Mode and any rule that challenges must not cover this name, since agents’ MCP clients, scripts and Let’s Encrypt cannot answer a challenge; Pseudo IPv4 and Rocket Loader stay off (their defaults).

Then, on the server, run aishie front-proxy.

The certificate

Caddy goes on getting its certificate from Let’s Encrypt itself, by the HTTP challenge: Let’s Encrypt asks http://HOST/.well-known/acme-challenge/…, Cloudflare in Full (strict) passes a plain HTTP request to the server’s port 80 as it is, and Caddy answers it. Caddy’s log may show a tls-alpn-01 challenge failing before an http-01 one passes, which is expected.

With Always Use HTTPS on, a server whose certificate is still valid can renew it, but a certificate that has run out, or a new server’s first, cannot be had: Full (strict) refuses a server with no valid certificate first (Cloudflare’s 526). Hence step 3 above: without it, one missed renewal takes the site down. aishie front-proxy says whether the path is right: goes through Cloudflare (cf-ray …) to Caddy, which answers it (308) is right; otherwise it warns what Cloudflare does instead (a 301 is Always Use HTTPS). It asks at the address public DNS gives the name, as Let’s Encrypt does. The certificate the server has now, and when it runs out:

echo | openssl s_client -connect 127.0.0.1:443 -servername aishie.example.edu 2>/dev/null | openssl x509 -noout -issuer -enddate

Do not use a Cloudflare Origin CA certificate: inside the server, the runtime reaches Core through Caddy at https://HOST and checks the certificate as a browser does, and an Origin CA certificate is trusted by Cloudflare alone.

Checking it

curl -sI https://aishie.example.edu/healthz | grep -i -e '^HTTP' -e '^server' -e '^cf-ray'

An answer of 200, server: cloudflare and a cf-ray line means the request went through Cloudflare. That is all it proves: whether visitors in mainland China reach the site has to be tested from a network there. aishie front-proxy says where Cloudflare’s addresses came from, whether Caddy took them, and what answers the certificate’s challenge path.

Limits of Cloudflare’s free plan

• A request’s body: 100 MB at most; a larger one is refused by Cloudflare (413). AIshie takes each file in a request of its own, of at most 50 MiB unless set (MAX_UPLOAD_BYTES), so this is never reached; only if MAX_UPLOAD_BYTES is raised past about 95 MiB does a file between the two fail in the browser, refused by Cloudflare. For larger files, keep it under that, or keep the files in a bucket (which browsers upload to directly, past Cloudflare).
• Waiting for an answer: about two minutes. The server must begin its answer within about 125 seconds, or Cloudflare answers 524. AIshie’s requests are all answered within that; the one exception is a large export of conversations, which Core takes up to 5 minutes to answer. An export that takes longer than two minutes gets a 524: narrow it (a course, a department, a participant or a span of time).

Nothing in the stack’s defaults goes past either limit, so nothing needs changing for Cloudflare.

Going back to direct connections

1. If FRONT_PROXY_ONLY is set, take it out of aishie.env first, and run aishie front-proxy.
2. In Cloudflare’s dashboard, set the record to DNS only (the grey cloud). Done before step 1, it has everyone who reaches the server directly refused until you run aishie front-proxy.
3. Keep FRONT_PROXY for five minutes at least: resolvers may still give Cloudflare’s addresses for the name (a proxied record’s TTL is 300 seconds), and the visitors they answer still come through Cloudflare. Then take it out and run aishie front-proxy, or leave it in: Caddy believes connections from Cloudflare’s addresses alone, so visitors who reach the server directly are not affected.

Rules

• One NAME=value per line, with no export and no comment after the value.
• Docker Compose reads the files. Values are used as they are (spaces included), but $ is taken as the start of a variable. Put a value that contains $ (such as a secret someone gives you) in single quotes: 'like$this'. FRAME_ANCESTORS in aishie.env is the only value in double quotes (see Letting other sites embed AIshie).
• After a change, recreate that service: aishie compose up -d core (or runtime, web). restart does not read the file again.
• stack.yaml sets the following from aishie.env, replacing whatever the files say:
  • Core: HTTP_ADDR=:8080, PUBLIC_URL=https://HOST, TRUSTED_PROXIES (Caddy’s address), RUNTIME_AUDIENCES=https://HOST/runtime.
  • The runtime: HTTP_ADDR=:9090, CONFIG=/config, SECRETS_DIR=/secrets, CORE_BASE_URL_ALLOWLIST, API_ADDR=:9091, API_AUDIENCE, CORE_BASE_URL, API_TRUSTED_PROXIES.
• aishie core help and aishie runtime help list every setting the running image reads, with its default.

aishie.env

setup-server.sh writes it once from its arguments; after that, you edit it by hand. It holds no secrets.

core.env

setup-server.sh writes DATABASE_URL, SIGNING_KEY, SECRETS_KEY and the file storage settings (BLOB_STORE, BLOB_FS_ROOT, and S3_* with a bucket). Everything else is optional; anything left unset takes Core’s default.

Keys

Files

Exporting conversations

Signing in and joining courses

Load and background jobs

Agents

Do not set INSECURE_COOKIES: it is only for development over http://localhost.

runtime.env

setup-server.sh writes DATABASE_URL (the runtime’s own database, not Core’s), LOG_FORMAT=json and KMS_KEY_ID=local:/secrets/kek/v1. Everything below is optional.

The settings below control scanned documents, Office files and transcription. /opt/aishie/env/runtime.env.example does not list them; aishie runtime help explains them in full. They are also the ceilings on what the site’s administrators set (see The operator’s ceilings).

postgres.env

POSTGRES_PASSWORD, AISHIE_CORE_DB_PASSWORD and AISHIE_RUNTIME_DB_PASSWORD. PostgreSQL reads them only when its data volume is first created; changing this file later does not change the passwords in the database (see Changing a database password).

Disk or bucket

• This server’s disk (BLOB_STORE=fs, the default): in /srv/aishie/core/blobs. Every upload and download goes through the server and uses its bandwidth; the disk has to grow with the files.
• A bucket (BLOB_STORE=s3): Core gives the browser a link it has signed, valid for a few minutes, and the browser uploads straight to the bucket and downloads straight from it. The files do not pass through the server, and the server’s disk does not hold them.

What to weigh

• Being near your users: browsers connect straight to the bucket’s region. A school in Hong Kong might choose AWS’s ap-east-1 (Hong Kong; the region must first be enabled in the AWS account) or ap-southeast-1 (Singapore), or an R2 bucket created with the “Asia-Pacific” location hint. B2 fixes the region when the account is opened, so make sure it is near your users. Core connects to the bucket only when attaching files (copying and deleting) and cleaning up, so a bucket far from the server only makes attaching files a little slower.
• Cost: the disk costs nothing beyond the server itself, but every byte uses the server’s bandwidth, and a bigger disk and its backups cost money too. Amazon S3 charges for storage, for each request, and for data going out to the internet (that is, every download). Cloudflare R2 charges for storage and for each operation; data going out is free. Backblaze B2 charges for storage, and downloads up to a multiple of the amount stored are free. For the numbers, see each provider’s price page for your region.
• The endpoint must be HTTPS: the site’s pages are https://, and browsers will not upload to http://.
• A bucket with dots in its name (such as files.example.edu) works only with path-style addressing, because no certificate covers https://the.bucket.name.endpoint. setup-server.sh refuses one together with --s3-path-style no, and refuses one in newer AWS regions that Core’s S3 client does not know. A bucket with no dots in its name works in any region.

S3_BUCKET_LOOKUP and S3_REGION

• S3_BUCKET_LOOKUP decides how a request names the bucket: path puts it after the endpoint (https://endpoint/bucket/key); dns puts it in the host name (https://bucket.endpoint/key), for services that accept only that; auto (the default) uses dns on AWS, Google and Alibaba Cloud, and path everywhere else.
• With --storage s3, --s3-path-style yes is written as path and no as dns; leaving it out gives auto. AWS, R2 and B2 always get auto.
• S3_REGION (default us-east-1; auto for R2) is the region requests are signed for; on AWS it is also the region they are sent to, including newer regions that Core’s S3 client does not know.
• Older Core images do not read S3_BUCKET_LOOKUP. When it is needed, setup-server.sh and aishie storage migrate first make sure the Core on the server knows it (aishie core help lists it).

Setting up a bucket

1. First create a private bucket at the provider (the links Core signs need no public access).
2. Make keys for it that can do only what Core does:
   • AWS: an IAM user or role allowed s3:ListBucket on the bucket, and s3:GetObject, s3:PutObject and s3:DeleteObject on the objects in it. Add s3:GetBucketCORS and s3:PutBucketCORS on the bucket, and setup-server.sh can set the CORS rule itself. In partitions other than aws, use that partition’s ARN, such as arn:aws-cn:s3::: in China.
   • R2: an R2 API token limited to this bucket, with Object Read & Write. The account ID is in the bucket’s S3 API address: https://<account>.r2.cloudflarestorage.com/<bucket>.
   • B2: an application key limited to this bucket, able to read and write. The bucket’s page shows its endpoint, s3.<region>.backblazeb2.com.
3. Run setup-server.sh and choose the bucket when it asks; or use options, with the keys in environment variables (see below).

Example: an IAM policy for the bucket aishie-files.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketCORS", "s3:PutBucketCORS"],
      "Resource": "arn:aws:s3:::aishie-files"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::aishie-files/*"
    }
  ]
}

When nobody is there to answer

The keys go only in the environment variables AISHIE_S3_ACCESS_KEY and AISHIE_S3_SECRET_KEY, never on the command line (command lines stay in the shell’s history). In bash:

read -r AISHIE_S3_ACCESS_KEY; read -rs AISHIE_S3_SECRET_KEY; export AISHIE_S3_ACCESS_KEY AISHIE_S3_SECRET_KEY
sh AIShie-Deploy/setup-server.sh aishie.example.edu edge --storage aws --s3-region ap-east-1 --s3-bucket aishie-files
sh AIShie-Deploy/setup-server.sh aishie.example.edu edge --storage r2 --r2-account-id <32 hex digits> --s3-bucket aishie-files
sh AIShie-Deploy/setup-server.sh aishie.example.edu edge --storage b2 --s3-region us-west-004 --s3-bucket aishie-files
sh AIShie-Deploy/setup-server.sh aishie.example.edu edge --storage s3 --s3-endpoint s3.example.com --s3-bucket aishie-files --s3-path-style no

(These are one example for each of the four services: run only one of the lines.) You can also use variables instead of options: AISHIE_STORAGE, AISHIE_S3_BUCKET, AISHIE_S3_REGION, AISHIE_S3_ENDPOINT, AISHIE_S3_PATH_STYLE, AISHIE_R2_ACCOUNT_ID and AISHIE_R2_JURISDICTION (eu or fedramp). With none of them, and nobody to answer, files go on the disk.

The checks

• Before writing anything, setup-server.sh checks the bucket with the keys, read-only: it lists at most one key, then asks for an object that does not exist (Core does the same when it starts). If the keys are refused, the region is wrong or the bucket does not exist, it stops, shows the service’s error code and message, and writes nothing.
• Since it writes nothing, it leaves nothing behind; only the first upload (or aishie storage migrate) proves the keys can write.
• You can check again at any time: aishie storage check.
• The keys never appear on the command line or in the output: core.env (0600) keeps them for Core.

The CORS rule

The site’s pages upload straight to the bucket, so the bucket must allow the site’s origin:

[
  {
    "AllowedOrigins": ["https://aishie.example.edu"],
    "AllowedMethods": ["GET", "HEAD", "PUT"],
    "AllowedHeaders": ["content-type"],
    "MaxAgeSeconds": 3600
  }
]

• When the bucket has no CORS rules and the keys are allowed to set them, setup-server.sh (and aishie storage migrate) sets this one itself. A bucket that already has rules is left alone, because that operation replaces every rule; instead, it prints the rule and where to set it.
• AWS console: S3 › Buckets › your bucket › Permissions › Cross-origin resource sharing (CORS) › Edit, paste it in (beside any rules already there), Save changes.
• Cloudflare dashboard: R2 Object Storage › your bucket › Settings › CORS Policy › Add CORS policy, paste it on the JSON tab, Save.
• Backblaze: the web console’s CORS settings offer only preset choices, so this rule has to be set through the S3 API, with a key that may change buckets (writeBuckets).
• Anywhere: aishie storage cors --apply --ask-keys sets it with keys you type when it asks (the keys are not kept). aishie storage cors checks whether the bucket already has the rule.

Without this rule, uploads on the site fail in the browser (the browser’s console mentions CORS). Downloads are links and need no rule.

Checking and moving: aishie storage

From the disk to a bucket

aishie storage migrate --to s3 --dry-run --storage aws --s3-region ap-east-1 --s3-bucket aishie-files
aishie storage migrate --to s3 --storage aws --s3-region ap-east-1 --s3-bucket aishie-files

The keys go in AISHIE_S3_ACCESS_KEY and AISHIE_S3_SECRET_KEY, or you type them when asked. --dry-run checks the bucket and says how many files there are, how many the bucket already has and how many it would copy, and changes nothing. The real move:

1. checks the bucket, and sets or checks the CORS rule (if there is no rule and it cannot set one, it stops before copying);
2. copies every file while Core keeps running, under the key Core reads it by, with the right content type;
3. compares every file on both sides by size and checksum (MD5 against the object’s ETag);
4. stops Core, copies the files uploaded during the copy, and compares again: the site is down for a few minutes;
5. writes BLOB_STORE=s3 and the bucket’s settings into core.env (the old file is kept as core.env.before-storage-<time>) and starts Core. If Core does not report healthy within a minute, it puts the old core.env back and starts Core again.

• It copies with rclone’s container (its version is pinned in aishie-storage), so nothing is installed on the server. If it stops partway or a comparison fails, it does not switch; running it again carries on from where it stopped.
• For a bucket whose ETags are not MD5 (such as one encrypted with SSE-KMS), use --size-only to compare sizes only.
• The copy on the disk stays, and Core no longer reads it. Once you are sure everything is fine, delete it with rm -r /srv/aishie/core/blobs.

What Core keeps

• documents/: the files of each version of course materials and assignments (one version can have several files).
• courses/: older uploads, from when a version had only one file.
• conversations/: files attached in conversations.
• renditions/: the PDFs that Office files were converted into.
• exports/: exported conversations, deleted after EXPORT_TTL.

With a bucket, files that have been attached are under attached/ (such as attached/documents/). Core ignores anything else in the directory or the bucket.

Two AIshie installations must never share a directory or a bucket: a test site made from a copy of the school site’s database, for example, would delete the other’s attached files as orphans.

What it does

• Hosts agents on AIshie: an agent’s owner hands the agent over on the site and chooses a model and key; the runtime is issued the agent’s one token by Core, by the agent’s ID, and keeps it encrypted, so nobody has to paste a token (see hosting an agent on AIshie in the User guide).
• Converts Office files to PDF: every Word, Excel, PowerPoint, OpenDocument and RTF file Core keeps is converted once into a PDF for previews on the site (see Office files shown as PDFs in the User guide). This uses no AI, costs nothing, and has no switch on the site. After an upgrade, existing files are converted newest first, and new uploads go ahead of them.
• Reads scanned documents (OCR): when a model cannot read a file itself, tesseract recognizes the text of scanned PDFs and images (Simplified Chinese, Traditional Chinese, English).
• LibreOffice: converts slides and documents into PDF for models that can read files, so that the model sees what the slides look like.
• Transcription (text versions): once an administrator turns it on, it transcribes course files into text versions with a model of the school AI plan (see text versions of files in the User guide). It uses a separate credential of its own, which a platform administrator issues on the site; that is not your job.
• Course materials search: agents can search a course’s materials while answering (see Course materials search).

The runtime’s credential in Core

In Core, the runtime is the site service agent_runtime, and it calls Core with its own credential (aissvc_…). Only with it can the runtime check agents’ owners, get and revoke agents’ tokens, and claim Office files to convert.

• The credential is in /etc/aishie/runtime/secrets/core/agent_runtime (0600, owned by the runtime’s user 65532), which the runtime reads as secret://core/agent_runtime. setup-server.sh issues it once Core has migrated.
• Without a valid credential, no agent runs (state error, reason runtime_misconfigured) and no Office file is converted. The agents’ own tokens are not affected.

Issuing a new credential

aishie runtime-credential

It asks Core to issue a new credential and revoke the service’s others, replaces the file whole (it is never half-written; if Core refuses, the file stays as it was), then recreates the runtime with the new credential. The agents the runtime hosts keep their tokens. On success, the last two lines are:

aishie: kept in /etc/aishie/runtime/secrets/core/agent_runtime, which the runtime reads as secret://core/agent_runtime; Core revoked any other the service held
aishie: the runtime recreated, with it

Run it when: the credential may have leaked; an administrator revoked it on the site; you put back /etc/aishie from a copy made before a rotation; or the runtime’s log says Core refused its credential (401).

Who can administer the runtime

The site’s AI and documents page (School AI plan, Pricing, Usage today, Documents, Agent hosting) is only for the runtime’s administrators. By default these are all of Core’s Root and platform administrators. To limit it to some of them, list their actor IDs in runtime.env (shown when you open the person’s page under Administration › People & agents), comma-separated:

ADMIN_ACTOR_IDS=<actor ID>,<actor ID>

Then run aishie compose up -d runtime. An account that is not Root or a platform administrator does not count, even if it is listed here. An administrator who is not on the list and opens the page sees “You are not one of this agent service’s administrators”.

The operator’s ceilings

On the site, the runtime’s administrators can turn OCR on and off, choose OCR’s languages, turn transcription on and off, add models to the school AI plan, and set quotas and prices (see AI and documents in the Administrator guide). Changes take effect within seconds, with no restart. But the following are yours to decide on the server, and the site cannot go beyond them:

• OCR=off: OCR never runs, whatever the site says. The other OCR_* settings exist only in runtime.env.
• TRANSCRIBE=off: transcription never runs.
• OFFICE_PDF=off: LibreOffice is not used, so Office files are not converted to PDF previews either. RENDITIONS=off stops only the conversion to PDF previews. With either, PDFs made earlier keep previewing, and a new Office file keeps showing “Converting to PDF…”; readers can still download it.
• The school: models in runtime.yaml and their key files: the site can see them but cannot change them, or add a model with the same ID.
• allowed_models and denied_models: models added on the site are limited by them too.
• KMS_KEY_ID: encrypts the school keys and owners’ keys that the site saves.

The quota and price settings made on the site replace those in runtime.yaml, until an administrator goes back to the server’s defaults (Use the server’s defaults).

Resources

• OCR: one CPU while it reads a file (at a lower priority than answering questions), and tesseract uses about 200 MB of memory per page. At the default 300 dpi, a 40-page scan takes a few minutes; OCR_DPI=200 is about twice as fast, but small print is harder to read.
• LibreOffice: one CPU and about 400 MB of memory while converting (at most 2 GiB). A 38-slide deck takes about 5 seconds.
• On a server with CPU to spare, you can raise OCR_CONCURRENCY or RENDITIONS_CONCURRENCY to 2.

Each answer’s budget

The runtime sets limits on every answer, so that one question cannot take too much time or money. The limits are under budgets.per_answer in runtime.yaml: set under runtime.defaults, they apply to every agent, those owners host on the site included; set in an agent YAML of your own, to that agent only.

The output limit of each single call to the model is another setting: the model’s params.max_output_tokens, 4000 by default; owners can set it for the agents they host under “Advanced” on the site (“Most output tokens per answer”, from 256 to 32000). For example, to let every agent write more in an answer, and take longer over it:

runtime:
  defaults:
    budgets:
      per_answer: {output_tokens: 24000, wall_clock_s: 240}

After changing it, run aishie runtime check --live, then aishie compose kill -s HUP runtime.

When the budget runs out: the forced last turn

• When the budget is nearly spent, the runtime forces a last turn: the model may call no more tools, and answers from what it has read. Every turn before it leaves it some output tokens: the least of 2,000, one call’s limit, and a quarter of output_tokens.
• A turn that hits its output limit before writing anything (a model that thinks may spend a whole turn thinking) forces the last turn too, rather than giving up at once.
• The forced turn, and a turn that carries on an answer that was cut short, ask the model for the least reasoning it takes: thinking off where it can be turned off (DeepSeek, GPT-5.5 and later, Gemini 2.5 Flash, for example), else the lowest reasoning effort; a model that already thinks least is asked nothing more; a family of models the runtime does not know (Kimi, GLM, Qwen on their own endpoints, for example) is sent no more than the reasoning effort configured. If the provider’s API refuses this with 400, the call is made again as configured, and the log says so. The same holds for models used through OpenRouter, Azure or Bedrock.
• A forced turn that writes nothing is asked once more for a short answer, of a few dozen to a few hundred words (asked_again in the answer’s log line).
• If there is still no answer, the person asking sees the notice for a spent budget (below).

A model that thinks counts its thinking as output tokens. If answers often end in the notice, or are cut short, raise output_tokens and the model’s params.max_output_tokens, or lower reasoning.effort where one is configured. The metric budget_exhausted_total{budget} counts which budget answers ran into; output_tokens includes a turn cut short before it wrote anything (see Metrics).

The notices’ language

When an agent cannot answer, it posts a notice in place of an answer: for a refusal (on_refusal_text), for a spent budget or every provider being unreachable (on_budget_text), and for a daily quota being used up (on_quota_text, or the school plan’s own). None of them is set by default, and the runtime posts its own notices:

• A refusal or a spent budget: in the language of the person asking: the agent’s fixed answer_language where it has one, else the language of the question’s text (English, Traditional Chinese or Simplified Chinese); where it cannot tell, in English and Traditional Chinese both.
• A quota used up: the runtime checks the quotas before it reads the conversation, so this notice cannot follow the question. It is in the language of the agent’s fixed answer_language where that is English or Chinese; otherwise it always comes in English and Traditional Chinese together, never in Simplified Chinese.

Which notice a used-up quota gives depends on the quota. An agent’s own daily budgets (budgets.per_agent_day and budgets.per_asker_day, or “Agents’ daily budgets” on the site) and a person’s quota on the school’s key (“Quotas per person” on the site) give the agent’s notice, on_quota_text. The school plan’s quotas (per_owner_day, per_asker_day and per_day under school:, or the plan’s “Daily quotas” on the site) give the plan’s own notice, set by on_quota_text under school:. When a quota on the school’s key is used up and the owner has put a key of their own behind the plan, that key answers and no notice is posted. The English notices are:

So with answer_language left to the asker, whoever asks once the school plan’s quota is used up reads, whatever language they asked in:

Today's school AI allowance is used up. Please try again tomorrow.

今天的學校 AI 額度已用完，請明天再試。

A text you set (in an agent’s settings, in runtime.defaults or under school:) is posted as written, whatever language the person asks in; leave it unset for the runtime’s own notices, in the languages above. examples/runtime.yaml in the AIShie-Agent-Runtime repository sets on_quota_text under runtime.defaults: if you copied it from there, the agents’ quota notice stays that sentence until you take it out. No sources are listed under a notice, and the site does not say “No course material cited” under it.

Checking the runtime

Example: a few lines of aishie runtime check’s output.

ocr: tesseract 5.5.0 chi_sim+chi_tra+eng 300dpi, 1 at once, 40 pages a file at most
office: LibreOffice 25.2.3.2, 1 at once, 300 pages a file at most, 2m0s a file
renditions: on, 1 at once, each in 5m0s at most, with LibreOffice 25.2.3.2, with the runtime's own credential in Core (CORE_SERVICE_CREDENTIAL)
transcriber: off in the site's settings; …

Agent states in /status worth watching for:

• error, reason runtime_misconfigured: the runtime has no valid credential; run aishie runtime-credential.
• error, reason mcp_agent: this is an agent with MCP access, which the runtime does not host.
• error, reason core_too_old: Core needs upgrading.
• unauthorized: the agent’s token was revoked in Core. An operator’s agent gets a new token when the runtime reads its settings again (HUP); for an agent hosted on AIshie, its owner asks for a new token on the site.
• owner_changed: the agent’s owner in Core is not the person who handed it to the runtime; it stops running until its owner hosts it again.

The operator’s own agents (optional)

An agent that you set up on the server, rather than one its owner hosts on the site, is a YAML file in /etc/aishie/runtime/agents/.

1. First create an agent hosted on AIshie on the site (or have an administrator register one), and note its ID.
2. Write the YAML, naming the agent by its agent_id. It holds no token: the runtime gets one with its own credential. The runtime’s docs/deploying.md and examples/ explain each setting.
3. Each secret it refers to (such as the model key secret://keys/tutor) is a file under /etc/aishie/runtime/secrets/ (here, keys/tutor). The YAML and the secrets belong to root and are readable by group 65532; the runtime cannot write to them.
4. Install, check, and put into effect:

   install -g 65532 -m 640 tutor.yaml /etc/aishie/runtime/agents/
   aishie runtime check --live
   aishie compose kill -s HUP runtime

This directory may hold at most one runtime: document (see The school AI plan). YAML files that are not an agent’s settings (such as the price list, prices.yaml) go in a subdirectory: a .yaml beside the agents is read as an agent’s settings.

Course materials search

An agent’s model can search a course’s materials with the course_materials_search tool.

• The index is kept in the runtime’s own database (the runtime’s migration 0014). It needs no PostgreSQL extension: the stack’s postgres:18 works as it is.
• It searches only what the asking seat may read: every answer checks with Core.
• Scans and older Office and OpenDocument files can be searched only once they have a text version, that is, with transcription on.
• A version’s index is deleted when Core purges that version, or after 30 days without use.

Its effect on Core’s call limit

The first search in each answer calls Core’s document_list once, then document_get once for each document listed (at most 100 times). At Core’s default RATE_LIMIT_PER_MINUTE (600 calls per account per minute), one agent can finish only about five such answers a minute in a course with many documents. When an agent is to answer many students in a large course, raise Core’s RATE_LIMIT_PER_MINUTE, and set polling.assumed_core_rate_per_min (default 600) in the runtime’s agent settings to the same number.

Outbound connections

• For agents hosted on AIshie, models are called only at the providers’ official endpoints; nobody can give the runtime a URL. When it connects, the runtime refuses any address that is not on the public internet (loopback, private networks, link-local and cloud metadata addresses, carrier-grade NAT, IPv6 forms that contain an IPv4 address, and other reserved ranges), and it does not follow redirects.
• With EGRESS_PROXY set, the runtime connects only to the proxy, which resolves the names and makes the connections, so the proxy itself must refuse these addresses.
• The school’s own model servers (a gateway, vLLM, Ollama) can only be offered through school: in runtime.yaml (see The school AI plan).
• The runtime waits for Core’s new messages with long polls of up to about 40 seconds each. Any proxy between the runtime and Core must allow requests at least 40 seconds.
• Any proxy or load balancer you add in front of the site must likewise allow requests at least 40 seconds (nginx’s default of 60 seconds is enough). For a large conversation export, Core keeps the request waiting up to 5 minutes: a proxy that cuts it sooner loses only that response, and the export itself still completes. The stack’s Caddy never cuts a request.

Three kinds of key

1. The owner’s own key: entered on the site for the agent, and kept encrypted in the runtime’s database under kek/v1 (see Rotating the runtime’s key).
2. The school AI plan (server): set in school: in runtime.yaml; each key is a file on the server, never stored in the database and never shown (see The school AI plan).
3. The school AI plan (site): added by the runtime’s administrators under AI and documents › School AI plan, with the provider’s official endpoint and the school’s key. It is tried before it is saved, encrypted like owners’ keys, and afterwards only a hint of it is shown (such as sk-…3f9a).

The providers you can choose on the site: OpenAI, Azure OpenAI, Anthropic, Google Gemini, Amazon Bedrock, DeepSeek, Qwen (Alibaba Cloud Model Studio), Moonshot (Kimi), Zhipu GLM and OpenRouter.

The school AI plan

A school can offer models on its own keys, so that people who host agents here do not have to bring their own. Without one, agents hosted on AIshie use only their owners’ keys.

1. Make the keys’ directory, write the key from standard input (paste the key, press Enter, then Ctrl-D), and install runtime.yaml:

   install -d -m 750 -g 65532 /etc/aishie/runtime/secrets/school /etc/aishie/runtime/secrets/school/keys
   install -m 640 -g 65532 /dev/stdin /etc/aishie/runtime/secrets/school/keys/anthropic-main
   install -g 65532 -m 640 runtime.yaml /etc/aishie/runtime/agents/
   aishie runtime check --live
   aishie compose kill -s HUP runtime

2. check lists the plan’s models and quotas; HUP puts them into effect. Once the runtime reports that it has a school key, the site offers School plan among the model choices for agents hosted on AIshie.

/opt/aishie has no example file; examples/runtime/runtime.yaml in the AIShie-Deploy repository is a place to start. Example (excerpt):

runtime:
  # prices_ref: prices/prices.yaml     # optional; needed only for quotas in dollars
  school:
    offers:
      - id: standard
        label: "School AI (Claude Haiku)"
        adapter: anthropic
        model: claude-haiku-4-5
        key_ref: secret://school/keys/anthropic-main
        params: {max_output_tokens: 1500}
    per_owner_day: {answers: 100}     # per person (all their agents together)
    per_asker_day: {answers: 20}      # per person asking (per agent, per course)
    # per_day: {answers: 5000}        # optional: a ceiling for the whole school

• The key file is 0640 root:65532; the directories school and school/keys are both 0750 root:65532. The key never leaves the server: the runtime reads it only when it calls the model.
• Quotas: per person per day, per_owner_day (100 answers when not set); per person asking, per_asker_day (20 when not set); and, optionally, per_day for the whole school. A day is counted in UTC and starts again at 00:00 UTC. An owner can put their own key behind the plan, which answers once the plan’s quota is used up; without one, the person asking is told the school’s allowance is used up for today (on_quota_text can reword this).
• Quotas in dollars (usd:) need a price list that prices every model, or the runtime does not start. The price list goes in a subdirectory, such as /etc/aishie/runtime/agents/prices/prices.yaml, named by prices_ref: prices/prices.yaml or by PRICES=/config/prices/prices.yaml in runtime.env. The numbers in the example examples/runtime/prices/prices.yaml only show the format: fill it in from the providers’ price pages.
• allowed_models and denied_models limit which models may use the school’s keys; models added on the site are limited by them too.
• Transcription’s model calls count toward the plan’s whole-school ceiling, not toward anyone’s quota.
• Each person’s use today: see the Usage today tab of the site’s AI and documents page.
• Changing a school key: replace the file (the runtime reads it only when it calls the model), then run aishie runtime check --live.
• When a model is removed from school:, agents that used it switch to the model and key their owner put behind it; without one, the agent stops and says the school has withdrawn the model.

Two kinds of provider

• The operator’s provider: set in OIDC_* in core.env; there can be at most one. The site’s Sign-in page lists it first, marked “Set by the server’s operator”, and it is read-only there.
• The site’s providers: Root and platform administrators add, test and switch them on under Administration › Sign-in (see single sign-on in the Administrator guide). Changes take effect at the next sign-in, with no restart. These providers need SECRETS_KEY.

Both kinds register the same redirect URI at the provider:

https://aishie.example.edu/v1/auth/sso/callback

Single sign-on does not create accounts: an administrator registers people first, and then either links their accounts (for the operator’s provider, and for site providers that do not link by email) or relies on a site provider that links by email (see linking accounts in the Administrator guide). Once single sign-on covers everyone, you can set JOIN_LINK_REGISTRATION=off in core.env to stop people signing themselves up through invitation links.

Setting up the operator’s provider

1. Register a client at the identity provider, with the address above as its redirect URI.
2. Write what it gives you into /etc/aishie/core.env:

   OIDC_ISSUER=https://adfs.example.edu/adfs
   OIDC_CLIENT_ID=<client ID>
   OIDC_CLIENT_SECRET=<client secret; in single quotes if it contains $>
   OIDC_PROVIDER_NAME=school-adfs
   OIDC_DISPLAY_NAME=示範大學 NetID

3. Run aishie compose up -d core. Core reads the provider’s discovery document when it starts.
4. Within a minute, the sign-in page shows Sign in with 示範大學 NetID.

• When OIDC_ISSUER has a value, the operator’s provider is on. For AD FS, the issuer is the AD FS server’s address followed by /adfs.
• OIDC_PROVIDER_NAME is the provider’s ID, and every account linked at it is recorded under this ID. Set it before anyone is linked, and never change it, or nobody already linked will be able to sign in. Left unset, it is the placeholder example-adfs: always set your own. If one of the site’s providers has the same ID, the operator’s provider takes its place.
• OIDC_DISPLAY_NAME is the name on the sign-in button: at most 64 printable characters, or Core does not start (aishie logs core says why). Left unset, the button uses the front end’s own words: Sign in with single sign-on.
• OIDC_SUBJECT_CLAIM (default upn) is the claim an account is known by; OIDC_SCOPES defaults to openid profile email. The defaults suit AD FS.
• The operator’s provider may be at any address: the limits on private addresses below do not apply to it.

SECRETS_KEY

The client secrets of the site’s providers are kept in Core’s database, encrypted with SECRETS_KEY from core.env (base64 of 32 random bytes). On a new server, setup-server.sh has already written it.

• Without it, the site cannot add providers (secrets_key_missing), and the page says the operator has to add it; the operator’s provider is not affected.
• What it encrypts, only it can open. Like SIGNING_KEY, copy it off the server (see What to keep off the server), never lose it, and change it only by rotating it (see Rotating SECRETS_KEY).
• If it is lost, or not rotated properly, the providers encrypted with it show “Secret can’t be opened” (secret_unavailable) and are no longer offered until an administrator enters their secret again.

Providers at private addresses

For the site’s providers, Core connects only to public addresses: when it reads the discovery document and keys, and when it calls the token endpoint at sign-in. Each time it connects, it checks the address it actually reaches, and refuses:

• loopback and localhost;
• private networks: 10/8, 172.16/12, 192.168/16, fc00::/7;
• link-local addresses: 169.254/16, fe80::/10 (where cloud metadata addresses are);
• the shared 100.64/10, multicast, and the other addresses IANA lists as not globally reachable, including these addresses written in IPv6.

Nor does it reach these providers through HTTPS_PROXY. The symptoms:

• an administrator presses Test, and the report gives the reason issuer_address_not_allowed;
• signing in through this provider fails (sso_provider_unavailable), and aishie logs core has the reason.

In these cases, add SSO_ALLOW_PRIVATE_ISSUERS=true to core.env, then run aishie compose up -d core:

• the school’s AD FS is on the campus network, and its name resolves to an address starting with 10.;
• the server can reach the internet only through a proxy;
• the server’s DNS answers every name with an address starting with 198.18. (fake-IP).

With it on, Core says so in its log when it starts; only an issuer on the same machine may use http.

If you remove the setting again, a site provider that is switched on and whose issuer is written as localhost or an address that is not public leaves the sign-in page; under Administration › Sign-in, its status is “Issuer not public” (issuer_address_not_allowed).

What the sign-in page sees

The front end’s image is the same for every server: Core decides which single sign-on buttons there are. The sign-in page asks Core’s GET /v1/auth/methods:

curl -s 127.0.0.1:8080/v1/auth/methods

With no single sign-on:

{"password": true, "password_accepts": ["login_id", "email"], "sso": null, "sso_providers": []}

With providers, sso_providers lists every one that people can sign in through now (the operator’s first), each with an id, a label (the name on the button) and a start. Browsers may keep the answer for a minute, so changes reach the sign-in page within a minute. When there is a provider people can sign in through, the sign-in page lists its buttons first, with the form for a student number and password behind the link below them.

Showing AIshie in a learning management system’s frame

1. Set FRAME_ANCESTORS in /etc/aishie/aishie.env, with the whole value in double quotes, because CSP’s keywords carry single quotes of their own:

   FRAME_ANCESTORS="'self' https://lms.example.edu"

2. Run aishie compose up -d web.
3. Check the header that browsers receive:

   curl -sI https://aishie.example.edu/ | grep -i content-security-policy

4. To stay signed in inside the frame, set COOKIE_SAMESITE=none in core.env, then run aishie compose up -d core: inside another site’s frame, Core’s sign-in cookie is a third-party cookie.

• FRAME_ANCESTORS="'none'" lets no page embed AIshie. Unset or empty, it means 'self'.
• A value written across several lines stops the front end from starting (aishie logs web has the reason), and every new front-end image after that is rolled back too, because its health check fails.
• Safari and every browser on iOS refuse third-party cookies, and so do browsers with third-party cookies turned off: there, AIshie can only be used in a tab of its own.
• Single sign-on inside a frame takes the frame to the provider’s sign-in page, and most providers do not allow themselves to be embedded.
• The other direction (AIshie showing another site in a frame, such as a similarity-checking service’s viewer) is not restricted.

The two channels

• edge: CORE_IMAGE, RUNTIME_IMAGE and WEB_IMAGE follow :edge, the newest version of each repository’s main that has passed its checks.
• stable: each channel must be a release, X.Y.Z, or a digest (sha256:…). On stable, aishie-update refuses tags that move by themselves, such as :edge, :1.3, :latest and :stable, so every upgrade, and the migrations that come with it, is someone’s decision.
• Each channel must be an image of its own repository: an image whose source label does not match is refused.

aishie-update

aishie-update.timer runs it every 5 minutes (with up to a minute’s random delay; runs missed while the server was off are made up when it starts). Each run first takes a lock, then deals with Core, the runtime and the front end in turn:

1. Pulls the channel’s tag and reads the digest it points to now.
2. The same as what is running: nothing to do. A digest that failed before: nothing either; it is logged once, until the channel points to another digest or someone runs --retry.
3. Any other digest is deployed:
   • Core: checks the image’s name and source label; reads its version; asks how far its migrations go (see migration 0027); backs up aishie_core to /var/backups/aishie/core-deploy-<UTC time>.dump (the last ten are kept; if the backup fails, it stops and changes nothing); runs migrate up and seed with the new image; recreates Core; within a minute, /healthz must report ok and the new version and commit.
   • The runtime: the same, except that before the backup it also checks the agents’ settings with the new image (check); the database is aishie_runtime; and there is only migrate up, no seed.
   • The front end: checks the name and label; recreates it; /version.json must report the commit on the image’s label.
4. If the new version does not report healthy, it switches back to the previous one (and waits for it to report healthy), and records the new digest as failed. The migrations were finished before the switch, and every migration leaves the previous version still working (except Core’s migration 0027).
5. After a deployment, it deletes the images of the three repositories that no container uses, and leaves other images alone.

A run stops at the first failure; until the recreate, the original version keeps running. The images that run are recorded by digest in /var/lib/aishie/images.env: a reboot or aishie compose up -d starts the same images, and does not follow the tags.

Example: the shape of aishie-update --status (only Core is shown; the runtime and the front end look the same).

aishie.example.edu (edge)

core
  running:     ghcr.io/aishie-education/aishie-core@sha256:…
  channel:     ghcr.io/aishie-education/aishie-core:edge
  last check:  2026-10-02T03:05:41Z up to date
  last result: 2026-10-02T02:40:12Z core sha256:… -> sha256:…: deployed … (…)

Upgrading stable

1. First try the version on an edge server (edge has run every release’s commit).
2. Set the new version in /etc/aishie/aishie.env on the stable server, such as CORE_IMAGE=ghcr.io/aishie-education/aishie-core:X.Y.Z.
3. Run aishie-update (or wait 5 minutes). It deploys Core first, then the runtime and the front end.
4. Look at aishie runtime-status: when it starts, the runtime reads Core’s tool catalogue, and refuses a Core in which tools it relies on are gone or have changed. If you upgraded only Core, restart the runtime (aishie compose restart runtime) so that it picks up what the new Core added to the tool catalogue.

To go back, set the previous version again and run aishie-update, or use aishie-update --pin (see Going back to the previous version).

Upgrading to 0.2.0

Update to 0.2.0 in this order: Core, then the agent runtime, then the front end, all three to 0.2.0.

1. Core first. The runtime and the front end were both checked against Core 0.2.0. An older Core refuses the front end’s requests for changes, and answers show no course materials they relied on.
2. Then the runtime. A runtime from before 0.2.0 never answers a question again once its answer has been sent back for changes, until the person asking writes again.
3. The front end last, and never before the runtime. Front end 0.2.0 offers Request changes on every proposal, agents’ answers in conversations included.

On a stable server, set CORE_IMAGE, RUNTIME_IMAGE and WEB_IMAGE in /etc/aishie/aishie.env to the three :0.2.0 images together, and run aishie-update: it deploys Core, the runtime and the front end in that order, and stops at the first failure. Its next run skips an image recorded as failed and goes on to deploy the services after it, so if Core or the runtime fails, set the channels after it back to 0.1.0 until the problem is solved (aishie-update --retry SERVICE tries again). A server that follows edge goes in the same order.

• Core’s migrations 0028 and 0029 and the runtime’s migration 0015 need nothing of you, and change no existing data; aishie-update runs migrate up before it switches over. There are no new settings.
• Going back to 0.1.0 means deploying the 0.1.0 images, with the database schema left as it is (see Going back to the previous version). When you put the runtime back to 0.1.0, put the front end back to 0.1.0 with it; until then, ask teachers to reject an agent’s answer with a reason rather than send it back for changes. Do not run migrate down (see Core’s migrations 0028 and 0029).

PostgreSQL

stack.yaml pins PostgreSQL 18, and aishie-update does not update it. For 18’s minor versions:

aishie compose pull postgres && aishie compose up -d postgres

A new major version cannot read the old one’s files: it takes a dump and restore by hand, with every service stopped. When AIShie-Deploy moves to a new major version, follow the “PostgreSQL's major version” section of /opt/aishie/README.md on the server.

Going back to the previous version

edge rolls back by itself: a new version that does not report healthy is switched back to the previous one. To roll back by hand, pin the service to an image; it is deployed with the same steps (backup, migrate up, switch, health check):

aishie-update --pin core sha256:<digest>      # the digest from the log or --status
aishie-update --pin core ghcr.io/aishie-education/aishie-core:X.Y.Z

• A pinned service no longer follows its channel, until aishie-update --unpin core.
• The new schema stays, and the previous version can run on it (except with Core’s migration 0027). To go back further than the previous version, restore a backup (see Restoring a backup).
• Do not run migrate down: it deletes data. The only exception is going back before Core’s migration 0027 (next section).

To see the schema’s version:

aishie core migrate version       # schema version N (embedded latest M)
aishie runtime migrate version

Core’s migration 0027

Migration 0027 removed columns that the previous version still reads and writes; it is the only migration that stops the previous version from working. A Core whose migrations stop before 0027 reports healthy on a database that has 0027, but every signed-in call that reads an account or a document version fails. A new server has 0027 from its first deployment, so this section matters only if someone pins or sets a Core image older than 0027.

Keep Core and the runtime on the same channel, with versions from the same time: a Core with 0027 requires the runtime to name a file when it tries the transcription credential, and an older runtime is refused (credential_rejected).

Before each Core image touches the database, aishie-update asks how far its migrations go, and protects the database in three cases:

• A new Core (with 0027) does not report healthy: it does not switch back to a Core from before 0027. The new one keeps running and is not recorded as failed, and the log says not rolled back: sha256:…'s migrations stop at 26, before Core's migration 0027. Look in aishie logs core for the cause (usually an env file), fix it, then run aishie compose up -d core.
• A Core from before 0027 is asked for (with --pin or by the channel): it is refused before the backup, nothing changes, the original Core keeps running, and the log says refused: sha256:…'s migrations stop at 26, before Core's migration 0027, which the schema has. While the channel points to it, every run refuses it (logged once), and the runtime and the front end update as usual.
• A new Core (with 0027) fails its seed: the previous Core keeps running, but the database now has 0027, so it cannot work properly. The log says so, and the new digest is recorded as failed. Fix what the seed points out and run aishie-update --retry core, or roll back by hand as below.

Going back before 0027 by hand (rare)

Stop Core, take the schema down one step with the image that has 0027 (the one running, which aishie core uses), then deploy the previous version. The site is down meanwhile, including while aishie-update makes its backup first:

aishie compose stop core
aishie core migrate down --yes      # says: schema version 26 (embedded latest 27)
aishie-update --pin core ghcr.io/aishie-education/aishie-core:<previous version>

On stable, the last step can instead be setting CORE_IMAGE back to the previous version and running aishie-update. Run migrate down only this once: running it again deletes data. Going down fills the data the previous version reads back in, from each version’s files and from the token of each agent hosted on AIshie, but the file types and sizes of versions that were purged cannot be filled back in.

After a failed seed, the Core running is already one from before 0027, and its migrate down cannot remove 0027 (no migration found for version 27). In that case, name the image that has 0027 by the digest in the log:

aishie compose stop core
CORE_REF=ghcr.io/aishie-education/aishie-core@sha256:<digest with 0027> \
  docker compose --project-directory /opt/aishie -f /opt/aishie/compose.yaml \
  run --rm --no-deps -T core migrate down --yes
aishie compose up -d core

Core’s migrations 0028 and 0029

These two migrations need nothing of you and change no existing data: aishie-update deploys them as usual. They come in the same Core release, and run in order. The previous release works on the new schema, so rolling back is still only a matter of going back to the previous image (see Going back to the previous version).

• 0028, requesting changes: whoever decides a proposal can send it back, saying what to change, and its proposer, proposing again, can name the proposal it revises (see Requesting changes in the User guide). After a rollback to the previous release, the site no longer requests changes, and shows proposals sent back for changes as over.
• 0029, the course materials an answer relied on: an agent’s answer can name the course materials it relied on, which Core keeps with the answer and shows to each reader as their permissions allow (see The course materials an answer relied on in the User guide). Until the agent runtime sends them, answers name none, as before. After a rollback to the previous release, the site neither writes nor reads sources; an answer proposed with sources that is still waiting for approval is then cancelled when it is approved, and the agent answers again.

Keep Core, the agent runtime and the front end on the same channel, with versions from the same time. Front end 0.2.0 offers Request changes on an agent’s answer too, and a runtime that does not know about requests for changes (one from before 0.2.0) never answers a question again once its answer has been sent back for changes; so update the runtime before the front end (see Upgrading to 0.2.0). The runtime’s own migration 0015 (which records answers sent back for changes) needs nothing of you either, and changes no existing data. Do not run migrate down for it: the runtime’s migrate down takes every migration down, deleting everything it keeps.

Do not run migrate down for these two migrations: taking 0029 down deletes every answer’s sources, and taking 0028 down turns every proposal sent back for changes into a rejection.

Core’s migrations 0030, 0031 and 0032, and the runtime’s migration 0016

These four migrations are on main after 0.2.0 (see Versions these guides describe). None needs anything of you: aishie-update deploys them as usual, Core’s three in order, then the runtime’s. The previous versions work on the new schemas, so rolling back is still only a matter of going back to the previous image (see Going back to the previous version).

• 0030, deleting an assignment for good: someone with the Manage assignments permission can delete an assignment for good, with its submissions, files and grades (see Deleting an assignment for good in the User guide). The migration adds the record of deletions, a queue of files waiting to be deleted, and a mark on the actions a deletion emptied; it changes no existing data. The database’s safeguards open only for deleting an assignment: submissions, grades, and the files of submissions and feedback are deleted only when an assignment is; materials, instructions and rubrics never are.
• 0031, group work: group sets, groups and who was in which group when, a group’s work and its grade (see Group work and peer evaluation in the User guide). It writes, for every existing submission, a record of whose work it is (that student’s), and points its grades at it.
• 0032, peer evaluation: the peer evaluation form, each evaluation and what it says, and a grade’s adjustment by peer evaluation. It changes no existing data.
• The runtime’s 0016, OpenRouter’s upstream routing: a model on the school’s plan can carry upstream routing for OpenRouter (see Upstream routing for an OpenRouter model in the Administrator guide). A runtime from before it does not read the column, and sends OpenRouter none of the routing the school set (it would not insist on upstream providers that keep no data, for example), so administrators should set upstream routing only once the runtime is updated. The stack has a single runtime container, which aishie-update replaces at once.

A large site: the indexes

0030 builds nine indexes, and 0031 builds one index of grades again. Building each one holds up writes to its table for a moment, which on a school’s site is only an instant. A site with a great deal of data can build them beforehand without holding anything up, and the migrations then skip them: Core’s docs/deploying.md (Migration 0030, Migration 0031) lists the CREATE INDEX CONCURRENTLY statements to run one at a time as the database’s owner, and how to check afterwards that no invalid index is left.

A deleted assignment’s files

Once an assignment is deleted, the files of its submissions and feedback are deleted from the file store (the disk or the bucket) the next time Core’s background jobs run (JOBS, every JOBS_INTERVAL). A file the store refuses to delete is tried again later (at most a day apart), and logged (queued file). If every Core has its background jobs turned off, they stay in the queue. Note: a database backup made before the deletion still has their records, but restoring it does not bring back files already deleted from the store (unless the bucket keeps old versions).

Going back, and migrate down

• When you go back to Core 0.2.0, take the front end back to 0.2.0 with it. 0.2.0 has no deleting of assignments, group work or peer evaluation; reading a group assignment that exists fails in it, while students’ own work and grades go on as usual. Updated again, everything is as it was.
• Do not run migrate down for these migrations. 0031’s down refuses to run while there is any group work, changes nothing, and says group work exists: delete the group assignments (assignment.delete) or restore from a backup taken before 0031; migrate version then stops at 30, marked dirty, and aishie core migrate force 31 sets it back. Even where it can run, it loses the group sets, groups and their history, and which assignments are group work. 0032’s down turns every peer adjustment into a grader’s plus or minus (the scores stay as they are), but loses the forms, every evaluation, the comments and the factors. 0030’s down loses the record of deletions and the queue of files waiting to be deleted.

When a migration fails

The log says migrate up failed, and the journal has the migration’s error. The original version keeps running, and the new digest is recorded as failed. A backup was made just before the failure (core-deploy-*.dump or runtime-deploy-*.dump).

Each migration runs in a transaction, so a failure usually leaves nothing behind, but the schema is marked dirty at the failed number N: migrate version shows DIRTY. Core’s /healthz answers 503 while the schema is dirty (the original version still serves requests); no version of the runtime can start on a dirty schema, but the one running carries on.

1. Fix the cause the error points to. Most often it is data the migration did not expect, and then a new image is needed, which the channel will move to by itself.
2. Record the migration before the one that failed as the last one done:

   aishie core migrate force <N - 1>
   aishie compose exec postgres psql -U postgres -d aishie_runtime \
     -c 'UPDATE schema_migrations SET version = <N - 1>, dirty = false'   # the runtime has no migrate force

3. Try again: aishie-update --retry core (or runtime).

If you are not sure what the failed migration left behind, restore the backup from before it instead.

Automatic backups

• Before every deployment of Core or the runtime: /var/backups/aishie/core-deploy-<time>.dump and runtime-deploy-<time>.dump, the last ten of each kept.
• Every night: at 03:00 server time (with up to 15 minutes’ random delay), aishie-backup.timer runs aishie backup, which writes core-daily-N.dump and runtime-daily-N.dump, N being the day of the week (1 to 7), so seven of each are kept.
• Back up now: aishie backup.

Each backup is a pg_dump -Fc run in the PostgreSQL container, written first as .part and renamed only once it is complete, readable by root only. If a backup fails, the one from the same day last week is still there; journalctl -u aishie-backup has the reason, usually a full disk.

Restoring a backup

Everything written after the backup is lost. Stop the service, rebuild the database from the backup, then pin the image that was running when the backup was made (the log records it):

aishie compose stop core
aishie compose exec postgres dropdb -U postgres aishie_core
aishie compose exec postgres createdb -U postgres -O aishie_core aishie_core
aishie compose exec -T postgres pg_restore -U postgres --exit-on-error --no-owner --role=aishie_core -d aishie_core < /var/backups/aishie/<file>.dump
aishie-update --pin core sha256:<the digest at the time>
aishie compose up -d core      # starts it again when that digest was already the deployed image

The runtime is the same, with core replaced by runtime and aishie_core by aishie_runtime. Answers the runtime has already posted to Core stay in Core.

If someone deleted an assignment for good after the backup was made, the restored database has the assignment and its submissions again, but their files may already have been deleted from the file store (see A deleted assignment’s files).

If the restored Core database is from before a SECRETS_KEY rotation, put the key from that time back in SECRETS_KEY_PREVIOUS and run aishie core secrets rewrap (see Rotating SECRETS_KEY). The runtime’s database likewise needs the kek key from its time.

What to keep off the server

Copy the following somewhere else, regularly:

1. /etc/aishie/, encrypted: SIGNING_KEY, SECRETS_KEY, the runtime’s key runtime/secrets/kek/v1, the database passwords, the runtime’s credential, and the model keys of the school plan and of the operator’s agents. No database backup can bring back SIGNING_KEY, SECRETS_KEY or the kek: without SECRETS_KEY, the secrets of the site’s single sign-on providers cannot be opened; without the kek, the keys the runtime keeps cannot be read. If the runtime’s credential is lost, simply issue a new one with aishie runtime-credential.
2. /var/backups/aishie/: the databases. Keep them apart from the copy of /etc/aishie: together, the two are every secret there is.
3. /srv/aishie/core/ (when files are kept on the disk): the uploaded files. Leave out /srv/aishie/core/blobs/exports/: exported conversations hold personal data, the server deletes them after EXPORT_TTL, and backups should not keep them any longer.
4. The bucket (with a bucket): the provider keeps it, and aishie backup does not include it. Turn on its versioning (or replication), and add a lifecycle rule that deletes old versions after a while, because Core deletes objects in its normal work too.

Copy /etc/aishie again after every rotation, and after setup-server.sh says it added a SECRETS_KEY or issued the runtime’s credential.

At a glance

Rotating SECRETS_KEY

SECRETS_KEY encrypts the client secrets of the site’s single sign-on providers, and only it can open them, so it is never simply replaced. If you suspect it has leaked:

1. In core.env, put the new key in SECRETS_KEY and the old one in SECRETS_KEY_PREVIOUS:

   SECRETS_KEY=<the new one: openssl rand -base64 32>
   SECRETS_KEY_PREVIOUS=<the old value of SECRETS_KEY>

2. Recreate Core with the new key, then encrypt every secret again with it:

   aishie compose up -d core      # Core encrypts with the new key; either key opens
   aishie core secrets rewrap     # encrypts every client secret again with the new key

   secrets rewrap says how many it encrypted again.
3. Once it has succeeded, delete the SECRETS_KEY_PREVIOUS line, run aishie compose up -d core, and copy /etc/aishie again.

• Until the rewrap is done, the site’s Sign-in page marks the providers still encrypted with the old key as “Older key”. The hint on the page says aishie-core secrets rewrap; in this stack, the command is aishie core secrets rewrap.
• If the rewrap fails, saying some provider’s secret opens with neither key: put the key that originally encrypted those secrets into SECRETS_KEY_PREVIOUS as well (several keys are comma-separated), recreate Core, and run it again; or have an administrator enter those providers’ secrets again on the site.
• Database backups from before the rotation are still encrypted with the old key. As long as you keep those backups, keep the old key separately too, because restoring one means putting it back in SECRETS_KEY_PREVIOUS and running the rewrap.
• A key that is not 32 bytes of base64 (part of it left out when pasting, say, or hex instead) stops Core from starting, and aishie logs core says which setting it is. The same happens with SECRETS_KEY_PREVIOUS but no SECRETS_KEY.

Rotating the runtime’s key (kek)

The runtime keeps owners’ model keys, school keys added on the site, and agents’ tokens encrypted in its database. Each secret has its own data key, and the keys in the kek/ directory wrap those data keys. The one that KMS_KEY_ID names encrypts new secrets; the other keys in the directory can still open what they encrypted (files whose names start with . do not count).

Check that every secret opens:

aishie runtime keys check

To change the key (when someone who could read it leaves, say, or on a schedule): add a new key, point KMS_KEY_ID at it, recreate the runtime, wrap every secret again with the new key, check, and only then remove the old key:

(umask 077 && openssl rand -base64 32 > /etc/aishie/runtime/secrets/kek/.v2.new)
chown root:65532 /etc/aishie/runtime/secrets/kek/.v2.new && chmod 640 /etc/aishie/runtime/secrets/kek/.v2.new
mv /etc/aishie/runtime/secrets/kek/.v2.new /etc/aishie/runtime/secrets/kek/v2
sed -i 's|^KMS_KEY_ID=.*|KMS_KEY_ID=local:/secrets/kek/v2|' /etc/aishie/runtime.env
aishie compose up -d runtime
aishie runtime keys rewrap
aishie runtime keys check      # every secret opens, and v2 wraps them all

• Keep v1 (and its copies) until every database backup older than the rewrap has been replaced (a week of nightly backups, and the last ten deployment backups): their secrets are still wrapped by v1. Only then run rm /etc/aishie/runtime/secrets/kek/v1.
• These steps follow the runtime’s own documentation, with this stack’s paths; AIShie-Deploy’s README does not include them yet. Before you do this for the first time, try it once on a server you can throw away.

Other secrets

The runtime’s credential

aishie runtime-credential, at any time (see The runtime’s credential in Core).

Database passwords

PostgreSQL reads postgres.env only when its data volume is created, so a new password has to be set both in the database and in the two files:

aishie compose exec postgres psql -U postgres -c '\password aishie_core'   # typed twice, not recorded

Then write it into AISHIE_CORE_DB_PASSWORD in postgres.env and into DATABASE_URL in core.env, and run aishie compose up -d core. The runtime is the same (aishie_runtime, AISHIE_RUNTIME_DB_PASSWORD, runtime.env). A hex password (openssl rand -hex 24) needs no escaping in the URL.

SIGNING_KEY

Do not change it in normal times: every upload and download link Core has issued, and every single sign-on in progress, stops working. ASSERTION_KEY, when it is not set, is derived from it and changes too. If it has leaked, change it anyway (write the output of openssl rand -hex 32 into core.env, then run aishie compose up -d core) and accept these effects.

Bucket keys

1. Make new keys at the provider, and write them into S3_ACCESS_KEY and S3_SECRET_KEY in core.env (a secret containing $ goes in single quotes).
2. Run aishie compose up -d core, then aishie storage check.
3. Delete the old keys. Links already issued and signed with the old keys stop working at once; they were valid for only a few minutes anyway.

School plan keys

Replace the files under /etc/aishie/runtime/secrets/school/keys/, then run aishie runtime check --live.

Logs

• aishie logs core (or runtime, web, caddy, postgres): shows the last 100 lines and keeps following. Any other options are passed straight to docker compose logs, such as aishie logs --since 1h core.
• Docker keeps at most 5 log files of 20 MB for each container.
• Core: one line of JSON per request.
• The runtime: one line of JSON per event, with only IDs, counts, outcomes and times; it never logs anything anyone wrote, or any token or key. Each OCR, conversion, PDF preview and transcription gets a line of its own. LOG_REDACT_EXTRA can remove more of what matches.
• Updates: /var/log/aishie-update.log has a line for each outcome (the time, the service, the digests before and after, the outcome); journalctl -u aishie-update has each step’s output and, when a health check fails, the last 30 log lines of the new version.
• Backups: journalctl -u aishie-backup.

Health checks

On the site, About AIshie in the account menu also shows the versions of the web app (the front end’s /version.json) and of the server (Core’s /healthz); one it cannot read says “Not known”. The sign-in page no longer shows the server’s version.

Metrics

Core has no metrics endpoint. The runtime serves metrics in Prometheus format at 127.0.0.1:9090/metrics, for a Prometheus on the same machine or to be read through an SSH tunnel; Caddy never routes it outside.

curl -s 127.0.0.1:9090/metrics

• The one most worth watching is presence_gap_seconds: above 60, Core shows the agent as away.
• long_poll_fallbacks_total{why="cut"} rising steadily means something between the runtime and Core is cutting long polls short (see Outbound connections).
• A large budget_exhausted_total means answers often reach the budget limit for a single answer; many with {budget="output_tokens"} usually mean a model that thinks has used up its output tokens (see Each answer’s budget).

Updates

• migrate up failed: see When a migration fails.
• not healthy within 60 seconds, followed by one of four:
  • rolled back: sha256:… runs again: the previous version is back and reports healthy, and the new digest is recorded as failed. The cause is in the new version’s last 30 log lines, in the journal.
  • not rolled back: …: the migration 0027 case; see Core’s migration 0027.
  • rolled back to sha256:…, which does not report healthy either: neither version comes up, most likely because of a setting they share: an env file, or the runtime’s agent settings. aishie logs core says which. Fix it, run aishie compose up -d core, then aishie-update --retry core to try the new version.
  • nothing ran before it, and nothing runs now: the service’s first deployment did not come up, and its container has been removed. Read the journal, fix the cause, then run aishie-update --retry SERVICE.
• refused: …'s migrations stop at 26: a Core from before 0027 was asked for; see Core’s migration 0027.
• skipped: it failed before: on purpose: a digest that failed is not retried every 5 minutes. When the channel points to a new digest, that is deployed as usual; or fix the cause on the server, then run aishie-update --retry SERVICE. A pinned service (--status shows pinned:) does not update until --unpin.
• could not pull …:
  • denied or unauthorized: the server still holds an expired ghcr.io login; run docker logout ghcr.io.
  • not found or manifest unknown: the channel in aishie.env points to an image or tag that is not published (or not yet), such as a release that is not out yet, or a typo.
  • Anything else (a timeout, connection refused, a TLS error): the server cannot reach ghcr.io (DNS, firewall, proxy), or GitHub’s service has a problem.
  After fixing it, run aishie-update.
• another aishie-update, or aishie backup, has held the lock: a deployment or backup is in progress, or an earlier run was interrupted and its process is still there: journalctl -u aishie-update -u aishie-backup, ps -ef | grep aishie.
• CORE_IMAGE refused: the channel is not an image of its own repository; or, on stable, it is not a release or a digest.
• PostgreSQL is not up and healthy: aishie logs postgres. Usually the disk is full: df -h /var/lib/docker /var/backups.
• the backup of the database … failed: nothing was changed, and the next run tries again. Usually the disk is full.
• could not pull the newest postgres:18 and caddy:2 (setup-server.sh): Docker Hub is limiting pulls for now (429 Too Many Requests). The setup carries on with the images the server already has; run it again later to get the newest.

The site does not open, or cannot be embedded

• Caddy has no certificate: aishie logs caddy. The DNS name must resolve to this server, and the provider’s firewall must allow 80 and 443 (ufw does not affect the ports Docker publishes).
• The learning management system cannot show AIshie in a frame: the browser’s console says the page refused to be embedded. Look at the header the front end sends with curl -sI https://aishie.example.edu/ | grep -i content-security-policy: the learning management system’s origin must be in it. If FRAME_ANCESTORS is not in double quotes, Docker cuts it at the first single quote and the header becomes frame-ancestors self, which no site can embed. After fixing it, run aishie compose up -d web.
• Signing in does not last inside the frame: set COOKIE_SAMESITE=none in core.env, then run aishie compose up -d core. Safari and browsers on iOS refuse it regardless.
• The front end does not start after a change to FRAME_ANCESTORS: aishie logs web has Caddy’s error; the value was written across several lines. Fix that line in aishie.env, then run aishie compose up -d web.

Single sign-on

• The sign-in page has no single sign-on button: curl -s 127.0.0.1:8080/v1/auth/methods. "sso": null means no provider is available: the core.env that Core runs with has no OIDC_ISSUER (after changing it, run aishie compose up -d core), and none of the site’s providers is switched on. The sign-in page may keep an old answer for a minute.
• The site cannot add a provider: secrets_key_missing: core.env has no SECRETS_KEY, or it has no value. If the line is missing, run setup-server.sh again (see Updating AIShie-Deploy’s own files): it adds one and recreates Core. If the line is there with no value, it leaves it alone: write the output of openssl rand -base64 32 there yourself, then run aishie compose up -d core. Either way, copy /etc/aishie again afterwards.
• A provider shows “Secret can’t be opened” (secret_unavailable): its secret was encrypted with a key Core no longer holds: the old key was deleted before the rewrap was done, a database from before a rotation was restored, or a core.env older than its SECRETS_KEY was put back. From your copy of /etc/aishie, put the key from that time into SECRETS_KEY_PREVIOUS, then run aishie compose up -d core and aishie core secrets rewrap. Without a copy, have an administrator enter the secret again on the site.
• Core does not start, and the log mentions SECRETS_KEY or SECRETS_KEY_PREVIOUS: a key is not 32 bytes of base64, or there is only SECRETS_KEY_PREVIOUS. Fix core.env, then run aishie compose up -d core.
• Core does not start, and the log mentions OIDC_DISPLAY_NAME: it is longer than 64 characters, or has characters that cannot be printed.
• The test reports issuer_address_not_allowed, or signing in fails with sso_provider_unavailable: the provider is at a private address, or the server reaches the internet through a proxy. See Providers at private addresses.
• Someone presses the button, and is told they are not registered yet: single sign-on does not create accounts. An administrator registers the person, then links their school account or uses a site provider that links by email.

Files and buckets

• Uploads fail in the browser, and the console mentions CORS: aishie storage cors checks whether the bucket has the rule the site needs, and prints the rule and where to set it; aishie storage cors --apply --ask-keys sets it with keys that are allowed to. The rule names https://HOST: update it after changing the server’s name.
• Core does not start: the S3 bucket is not reachable: Core asks the bucket for an object when it starts. aishie storage check connects with the keys in core.env and says why it was refused: keys deleted or mistyped, the wrong region, a bucket that does not exist. Fix core.env, then run aishie compose up -d core.
• needs a Core whose help names S3_BUCKET_LOOKUP (setup-server.sh or aishie storage migrate): the Core on the server is too old to reach the bucket with --s3-path-style no or in a newer AWS region; nothing was changed. Update Core first, then run it again.
• aishie storage migrate stopped: it did not switch, core.env is unchanged, and the Core it stopped has been started again. Fix what it points out and run it again; it copies only the files not there yet. If it reports the two sides differ for every file and the bucket is encrypted with SSE-KMS, add --size-only. could not pull rclone/rclone: Docker Hub is limiting pulls for now; try again later.
• Core did not report healthy on the other side: core.env has been put back and Core started with it; nothing is lost, and the files are on both sides. aishie logs core says why Core could not start on the other side.

The agent runtime

• The AI and documents page says “This server has no agent service” or “The agent service cannot be reached”: aishie ps shows whether runtime is running, aishie logs runtime gives the reason, and curl -s https://aishie.example.edu/runtime/api/v1/info confirms the API can be reached through Caddy. While the runtime restarts, it cannot be reached for a moment.
• “You are not one of this agent service’s administrators”: ADMIN_ACTOR_IDS in runtime.env does not list this administrator (see Who can administer the runtime).
• The runtime hosts no agents, and its log says Core refused its credential (401): the credential was revoked (by an administrator on the site, or by putting back an /etc/aishie from before a rotation). Run aishie runtime-credential. Do not edit or empty that file by hand.
• the runtime was not given its credential for Core (setup-server.sh) or Core issued no credential (aishie runtime-credential): Core is not deployed yet. Once aishie-update --status shows Core running, run aishie runtime-credential.
• The runtime’s check refused the settings: the new version does not accept the current agent YAML. aishie runtime check gives the reason with the running version, and the journal has what the new version says. Fix the YAML (so that both versions accept it), run aishie compose kill -s HUP runtime, then aishie-update --retry runtime.
• Transcription shows Blocked: …: for “no credential” or “AIshie refused its credential”, a platform administrator issues the transcription credential again on the site’s Documents tab; “no model chosen”, “its model is not available” and “today’s pages are used up” are dealt with there too. If the page says the server has turned transcription or OCR off (its ⓘ says “TRANSCRIBE=off in the agent service’s environment.” or “OCR=off in the agent service’s environment.”), that is TRANSCRIBE=off or OCR=off in runtime.env.
• An agent shows as Paused in the chat window, or only as “Last connected …” on its page: presence_gap_seconds is above 60. Look at aishie runtime-status and aishie logs runtime.
• An agent often answers “I couldn’t finish this one. Try a narrower question.”: answers are using up their budget, most often a model that thinks using up its output tokens. Look at budget_exhausted_total{budget}, then raise budgets.per_answer.output_tokens and params.max_output_tokens, or lower the reasoning effort (see Each answer’s budget).

Office files keep showing “Converting to PDF…”

Users are told to let an administrator know (see the User guide). The runtime does the converting, so:

1. aishie ps: is the runtime running?
2. The renditions: line of aishie runtime check: is it on, or does it say why it cannot be (no credential, say, or OFFICE_PDF=off or RENDITIONS=off)?
3. Without a valid credential, nothing is converted: aishie runtime-credential.
4. The metrics: is rendition_jobs_total going up, and does rendition_claim_errors_total show errors?
5. Just after a Core upgrade, existing Office files are converted one by one (one at a time by default), newest first, with new uploads ahead of them, so older files take a while.

When a conversion fails or is skipped, the file can only be downloaded. The reasons:

Teaching staff can send a failed file for conversion again on the site.

Limits

• An upload is refused for its size: a single file is limited by MAX_UPLOAD_BYTES; a version of course material by DOCUMENT_MAX_FILES_PER_VERSION and DOCUMENT_MAX_VERSION_BYTES; files in conversations by ATTACHMENT_MAX_BYTES, ATTACHMENT_MAX_PER_MESSAGE and ATTACHMENT_MAX_CONVERSATION_BYTES (see core.env).
• A conversation export is refused: export_too_large: ask the administrator to narrow it down, or raise EXPORT_MAX_MESSAGES or EXPORT_MAX_BYTES.
• A large export gets no response: a proxy of your own in front of the site cut the request within 5 minutes. The export itself still completes; only this response was lost. Set the proxy’s time limit above 5 minutes.
• Agents answer slowly, and core_calls_total shows many rate_limited: agents searching a lot of course material have used up Core’s call limit. See Course materials search.

Commands

All are run on the server as root. aishie core … and aishie runtime … run in a one-off container, with the running image, its env file, its mounts and its network.

Ports