CI/CD Workflows

CI/CD Workflows

Back to Home

This document describes the GitHub Actions workflows that automate building worker binaries and deploying the ByteBiota server. These workflows provide automated CI/CD capabilities for both development and production releases.

Overview {#overview}

ByteBiota uses two main GitHub Actions workflows to automate the build and deployment process:

  1. Build Worker Workflow - Creates cross-platform worker binaries for distribution
  2. Deploy Workflow - Deploys the server to production environments

Both workflows support both automatic triggers (via git tags) and manual dispatch for flexibility in development and release processes.

Build Worker Workflow {#build-worker-workflow}

File: .github/workflows/build-worker.yml
Purpose: Builds standalone worker binaries for Linux, macOS, and Windows platforms

Triggers {#build-worker-triggers}

The workflow runs automatically on:

  • Tag pushes matching patterns:
  • worker-v* (e.g., worker-v2025.01.15)
  • worker/v* (e.g., worker/v2025.01.15)

  • v* (e.g., v2025.01.15)

  • Manual dispatch via GitHub Actions UI with optional inputs:

  • version: Override the worker version (e.g., 2025.02.14.1)
  • server_url: Server URL baked into the binary (default: https://www.bytebiota.com)

Build Matrix {#build-matrix}

The workflow builds for multiple platforms in parallel:

Platform OS Architecture Python Version
Linux ubuntu-latest amd64 3.11
macOS macos-latest amd64 3.11
Windows windows-latest amd64 3.11

Build Process {#build-process}

Each platform build follows these steps:

  1. Checkout code - Gets the latest source code
  2. Setup Python - Installs the specified Python version
  3. Cache dependencies - Caches pip downloads for faster builds
  4. Install dependencies - Installs requirements.txt and PyInstaller
  5. Resolve metadata - Determines version and server URL from inputs or git tags
  6. Build binary - Runs python scripts/build_worker.py --profile release
  7. Upload artifacts - Uploads built binaries and archives

Version Resolution {#version-resolution}

The workflow automatically determines the version using this priority:

  1. Manual input - If version is provided via workflow_dispatch
  2. Git tag parsing - Extracts version from tag name:
  3. worker-v1.2.3 → 1.2.3
  4. worker/v1.2.3 → 1.2.3
  5. v1.2.3 → 1.2.3
  6. Default - Falls back to empty string if no version found

Server URL Configuration {#server-url-config}

The server URL is determined by:

  1. Manual input - If server_url is provided via workflow_dispatch
  2. Default - Uses https://www.bytebiota.com

Output Artifacts {#build-artifacts}

Each successful build produces:

  • Binary distribution in dist/bytebiota-worker-{version}-{platform}-{arch}/
  • Archive files in releases/:
  • Linux/macOS: bytebiota-worker-{version}-{platform}-{arch}.tar.gz
  • Windows: bytebiota-worker-{version}-windows-amd64.zip
  • Checksums for integrity verification

Artifacts are uploaded to GitHub Actions and available for download for 90 days.

Deploy Workflow {#deploy-workflow}

File: .github/workflows/deploy.yml
Purpose: Deploys the ByteBiota server to production environments

Triggers {#deploy-triggers}

The workflow runs automatically on:

  • Tag pushes matching patterns:
  • release/server-* (e.g., release/server-2025.01.15)

  • server-v* (e.g., server-v2025.01.15)

  • Manual dispatch via GitHub Actions UI with optional input:

  • ref: Git reference (branch or tag) to deploy (defaults to current ref)

Required Configuration {#deploy-config}

The workflow requires these GitHub repository secrets:

Secret Description Example
DEPLOY_KEY SSH private key for server access -----BEGIN OPENSSH PRIVATE KEY-----...
DEPLOY_USER Username for SSH connection bytebiota
DEPLOY_HOST Server hostname or IP address prod.bytebiota.com

And this repository variable:

Variable Description Example
DEPLOY_PATH Target directory on server /opt/bytebiota

Deployment Process {#deployment-process}

The deployment follows these steps:

  1. Checkout code - Gets the specified git reference
  2. SSH setup - Configures SSH key and known hosts
  3. Sync code - Uses rsync to transfer files to server
  4. Install dependencies - Sets up Python virtual environment and installs packages
  5. Restart service - Restarts the systemd service

Code Synchronization {#code-sync}

The rsync command excludes these directories/files:

  • .git - Git repository data
  • .github - GitHub Actions workflows
  • __pycache__ - Python cache files
  • .env* - Environment files
  • *.env - Environment files
  • data/ - Runtime data
  • logs/ - Log files
  • *.pyc - Python bytecode
  • .pytest_cache - Test cache
  • *.egg-info - Python package info
  • releases/ - Release artifacts
  • checkpoints/ - Checkpoint data

Service Management {#service-management}

The deployment assumes a systemd service named bytebiota is configured on the target server. The workflow:

  1. Installs dependencies in a virtual environment
  2. Installs the package in development mode (pip install -e .)
  3. Restarts the service with sudo systemctl restart bytebiota

Usage Examples {#usage-examples}

Triggering Worker Builds {#trigger-worker-builds}

Automatic Build via Tag
# Create and push a tag to trigger automatic build
git tag worker-v2025.01.15
git push origin worker-v2025.01.15

# Or use a generic version tag
git tag v2025.01.15
git push origin v2025.01.15
Manual Build with Custom Parameters
  1. Go to GitHub Actions → "Build worker artifacts"
  2. Click "Run workflow"
  3. Fill in optional parameters:
  4. Version: 2025.01.15.1
  5. Server URL: https://staging.bytebiota.com
  6. Click "Run workflow"

Triggering Server Deployments {#trigger-server-deployments}

Automatic Deployment via Tag
# Create and push a server release tag
git tag release/server-2025.01.15
git push origin release/server-2025.01.15

# Or use server version tag
git tag server-v2025.01.15
git push origin server-v2025.01.15
Manual Deployment of Specific Branch
  1. Go to GitHub Actions → "Deploy ByteBiota"
  2. Click "Run workflow"
  3. Fill in optional parameter:
  4. Ref: feature/new-feature (branch name) or commit-hash
  5. Click "Run workflow"

Downloading Build Artifacts {#download-artifacts}

  1. Go to GitHub Actions → "Build worker artifacts"
  2. Click on a successful workflow run
  3. Scroll to "Artifacts" section
  4. Download the platform-specific archive:
  5. bytebiota-worker-linux-amd64 for Linux
  6. bytebiota-worker-macos-amd64 for macOS
  7. bytebiota-worker-windows-amd64 for Windows

Troubleshooting {#troubleshooting}

Build Failures {#build-failures}

Common issues:

  1. Dependency installation fails
  2. Check that requirements.txt is valid

  3. Verify PyInstaller compatibility with Python 3.11

  4. Version resolution fails

  5. Ensure tag names match expected patterns

  6. Check that manual version input is valid

  7. Platform-specific build errors

  8. Review build logs for platform-specific issues
  9. Test build script locally on target platform

Deployment Failures {#deployment-failures}

Common issues:

  1. SSH connection fails
  2. Verify DEPLOY_KEY secret is correctly formatted
  3. Check that DEPLOY_HOST is accessible

  4. Ensure SSH key has proper permissions

  5. Permission denied errors

  6. Verify DEPLOY_USER has sudo access for service restart

  7. Check that DEPLOY_PATH is writable by the user

  8. Service restart fails

  9. Ensure systemd service bytebiota exists
  10. Check service configuration and dependencies
  11. Verify virtual environment setup

Debugging Workflows {#debugging}

  1. Enable debug logging - Add ACTIONS_STEP_DEBUG: true to workflow secrets
  2. Check workflow logs - Review detailed logs in GitHub Actions UI
  3. Test locally - Run build script locally to isolate issues
  4. Verify secrets - Ensure all required secrets are properly configured

Security Considerations {#security}

Build Security {#build-security}

  • No secrets in build - Worker binaries don't contain sensitive information
  • Version verification - Use checksums to verify binary integrity
  • Source validation - Only build from trusted git references

Deployment Security {#deployment-security}

  • SSH key management - Use dedicated deployment keys with minimal permissions
  • Network security - Deploy over secure connections (SSH)
  • Access control - Limit deployment user permissions to necessary operations
  • Audit trail - GitHub Actions provides complete deployment history

Best Practices {#best-practices}

Tagging Strategy {#tagging-strategy}

  • Worker releases: Use worker-v{version} for worker-specific releases
  • Server releases: Use release/server-{version} for server deployments
  • Generic releases: Use v{version} for combined releases
  • Semantic versioning: Follow semantic versioning (e.g., 2025.01.15)

Release Process {#release-process}

  1. Test locally - Verify builds work on target platforms
  2. Create release branch - Use dedicated branches for releases
  3. Tag appropriately - Use descriptive tag names
  4. Monitor deployment - Watch workflow execution and logs
  5. Verify deployment - Test deployed services after completion

Maintenance {#maintenance}

  • Regular updates - Keep GitHub Actions and dependencies updated
  • Monitor usage - Track runner minutes and optimize workflows
  • Documentation - Keep this documentation current with workflow changes
  • Testing - Test workflow changes in development before production use