Worker Build Process

Worker Build Process

Back to Home

This document describes the ByteBiota worker build process, including how to create standalone binaries with custom server configurations.

Overview {#overview}

The ByteBiota worker build process creates standalone, self-contained binaries that can run on any compatible system without requiring Python installation. The build process automatically configures the worker to connect to a specified server URL with full SSL/TLS support.

Build scripts {#build-scripts}

ByteBiota now ships a single, cross-platform build orchestrator with thin wrappers for each operating system:

Python build orchestrator

scripts/build_worker.py performs all heavy lifting:

  • Detects the current platform (Linux, macOS, Windows) and CPU architecture (amd64, arm64).
  • Supports multiple build profiles (release and dev) with sensible defaults for each workflow.
  • Generates date-based version strings for release builds and applies dev-YYYYMMDD-HHMMSS markers for local builds.
  • Injects the build version and server URL into the packaged binary via a PyInstaller runtime hook.
  • Moves PyInstaller output to dist/bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}.
  • Produces checksums and platform-appropriate archives (.tar.gz on Unix, .zip on Windows).

Run it directly with the system Python interpreter:

python3 scripts/build_worker.py

Build profiles {#build-profiles}

The orchestrator exposes two profiles, selectable via --profile or the WORKER_BUILD_PROFILE environment variable:

Profile Purpose Behaviour
release (default) Production artifacts Auto-increments date-based versions, compresses Unix binaries, generates SHA-256 checksums, writes archives to releases/, and requires a clean git tree.
dev Local testing Generates timestamped dev- versions, keeps PyInstaller build directories, skips gzip compression and checksum generation, and stores outputs under dist/dev/.

Examples:

# Local iteration (keeps build artifacts, no archives)
python3 scripts/build_worker.py --profile dev

# Force a release-style build locally
python3 scripts/build_worker.py --profile release

# Set default profile through the environment
export WORKER_BUILD_PROFILE=dev
python3 scripts/build_worker.py

Configuration {#configuration}

Default Server URL

By default, workers are configured to connect to https://www.bytebiota.com:

Python (all platforms):

python3 scripts/build_worker.py

All platforms:

# Unix/macOS/Linux
python3 scripts/build_worker.py

# Windows
py -3 scripts\build_worker.py

Custom Server URL

To build a worker for a different server:

Python (all platforms):

# Custom server URL
SERVER_URL=https://your-server.com python3 scripts/build_worker.py

# Local development server
SERVER_URL=http://localhost:8080 python3 scripts/build_worker.py

Environment variables work the same way on all platforms:

  • Unix: SERVER_URL=https://your-server.com python3 scripts/build_worker.py
  • Windows: $env:SERVER_URL = "https://your-server.com"; py -3 scripts\build_worker.py

Version Control

Python (all platforms):

VERSION=2025.01.15 python3 scripts/build_worker.py

Version control works the same way on all platforms:

  • Unix: VERSION=2025.01.15 python3 scripts/build_worker.py
  • Windows: $env:VERSION = "2025.01.15"; py -3 scripts\build_worker.py

Automatic version detection

Release builds automatically detect existing archives in releases/ and increment date-based versions:

  • First build of the day: Uses base version (e.g., 2025.01.15)
  • Subsequent builds: Auto-increments (e.g., 2025.01.15.1, 2025.01.15.2)
  • Cross-platform: Each platform maintains its own version sequence

Development builds default to a dev-YYYYMMDD-HHMMSS stamp. Provide --version if you want a stable identifier while iterating.

Release profile builds abort if the git working tree is dirty or if a dev- version is requested, preventing accidental publication from local changes.

Prerequisites {#prerequisites}

  • Python 3.9+ with PyInstaller
  • Git (optional, for version control)
  • Platform-specific tools: tar/gzip (Unix), zip (Windows)

Build Output {#build-output}

The build process creates several artifacts:

Binary distribution

dist/bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}/
├── bytebiota-worker[.exe|.gz]   # Main executable (gzip on Unix release builds)
├── worker.conf                  # Default configuration
└── bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}.sha256  # Checksum (release profile)

Development builds live under dist/dev/ and omit the checksum file unless you pass --archive.

Archive files

releases/
├── bytebiota-worker-{VERSION}-{PLATFORM}-{ARCH}.tar.gz  # Linux/macOS
└── bytebiota-worker-{VERSION}-windows-amd64.zip         # Windows

Archives are only produced in the release profile unless you explicitly request them via --archive.

Continuous integration {#continuous-integration}

GitHub Actions publishes release artifacts through .github/workflows/build-worker.yml:

  • Triggers: tag pushes (v*, worker-v*, worker/v*) and manual workflow_dispatch runs.
  • Matrix: Linux, macOS, and Windows runners with Python 3.11.
  • Behaviour: Runs the release profile, uploads dist/ outputs plus the packaged archives from releases/.
  • Inputs: Manual runs can override the baked version (version) and server URL (server_url).

Use manual dispatches for staging builds while keeping production releases gated behind signed tags.

Server deployments follow a separate workflow. .github/workflows/deploy.yml only executes for tags that match release/server-* or server-v*, or when launched manually with a custom ref, so routine commits do not consume runner minutes on private plans.

For complete CI/CD workflow documentation, including detailed triggers, inputs, outputs, troubleshooting, and usage examples, see CI/CD Workflows.

SSL/TLS Support {#ssl-support}

The worker automatically supports secure connections:

HTTPS Support

  • HTTP API calls use HTTPS when server URL starts with https://
  • SSL certificate validation is enabled by default
  • Supports modern TLS protocols (1.2, 1.3)

WebSocket SSL (WSS)

  • WebSocket connections automatically use WSS for HTTPS servers
  • URL conversion: https://server.com → wss://server.com
  • Trailing slashes are normalized to prevent double slashes in WebSocket paths
  • Maintains secure connection throughout worker lifecycle

Configuration

# Secure connection (automatic SSL/TLS)
SERVER_URL=https://secure-server.com python3 scripts/build_worker.py

# Insecure connection (development only)
SERVER_URL=http://localhost:8080 python3 scripts/build_worker.py

Build Process Details {#build-details}

Environment Variables

The build script uses environment variables to configure the worker binary without modifying source files:

# Set during build (handled automatically by scripts/build_worker.py)
export WORKER_VERSION="$VERSION"
export SERVER_URL="$SERVER_URL"

# worker_entry.py reads from environment variables
BUILD_VERSION = os.environ.get('WORKER_VERSION', '0.1.0')
BUILD_SERVER_URL = os.environ.get('SERVER_URL', 'http://localhost:8080')

Clean Build Process

The build process is designed to be clean and reproducible:

  • No file modifications: The build script never modifies source files
  • Environment-based configuration: All build-time values are passed via environment variables
  • Template-based approach: worker_entry.py serves as a template that reads from environment variables
  • Runtime hook injection: scripts/build_worker.py generates a temporary PyInstaller runtime hook to bake WORKER_VERSION and SERVER_URL into the final executable
  • Reproducible builds: The same source code produces consistent results across different machines

PyInstaller Configuration

The build uses worker.spec for PyInstaller configuration:

  • Includes all required dependencies
  • Creates single-file executable
  • Handles platform-specific requirements
  • Optimizes for size and performance

Platform Support

Platform Architecture Binary Format Archive Format
Linux amd64, arm64 ELF .tar.gz
macOS amd64, arm64 Mach-O .tar.gz
Windows amd64 PE .zip

Usage Examples {#usage-examples}

Basic build

macOS/Linux:

# Build for current platform
python3 scripts/build_worker.py

# Output: dist/bytebiota-worker-2025.01.15-macos-amd64/

Windows (PowerShell):

# Build for current platform using the Python launcher
py -3 scripts\build_worker.py

# Output: dist\bytebiota-worker-2025.01.15-windows-amd64\

Production build

macOS/Linux:

# Production server with custom version
VERSION=2025.01.15 SERVER_URL=https://prod.bytebiota.com python3 scripts/build_worker.py

# Verify build
cd dist/bytebiota-worker-2025.01.15-macos-amd64/
./bytebiota-worker --help
./bytebiota-worker --server-url https://prod.bytebiota.com

Windows (PowerShell):

# Production server with custom version
$env:VERSION = "2025.01.15"
$env:SERVER_URL = "https://prod.bytebiota.com"
py -3 scripts\build_worker.py

# Verify build
cd dist\bytebiota-worker-2025.01.15-windows-amd64\
./bytebiota-worker.exe --help
./bytebiota-worker.exe --server-url https://prod.bytebiota.com

Development build

macOS/Linux:

# Local development server
SERVER_URL=http://localhost:8080 python3 scripts/build_worker.py

# Test locally
cd dist/bytebiota-worker-2025.01.15-macos-amd64/
./bytebiota-worker --server-url http://localhost:8080 --verbose

Windows (PowerShell):

# Local development server
$env:SERVER_URL = "http://localhost:8080"
py -3 scripts\build_worker.py

# Test locally
cd dist\bytebiota-worker-2025.01.15-windows-amd64\
./bytebiota-worker.exe --server-url http://localhost:8080 --verbose

Troubleshooting {#troubleshooting}

Build Failures

# Check dependencies
pip install -r requirements.txt pyinstaller

# Clean previous builds
rm -rf build/ dist/

# Verbose build
python3 scripts/build_worker.py --keep-build

SSL Issues

# Test SSL connection
curl -I https://www.bytebiota.com

# Check certificate
openssl s_client -connect www.bytebiota.com:443 -servername www.bytebiota.com

Platform Issues

# Check platform detection
uname -s  # Should return Linux, Darwin, or CYGWIN/MSYS
uname -m  # Should return x86_64, amd64, or arm64

Best Practices {#best-practices}

Security

  • Always use HTTPS for production deployments
  • Verify SSL certificates are valid
  • Use strong server authentication

Performance

  • Build on target platform when possible
  • Test binaries before distribution
  • Monitor resource usage in production

Maintenance

  • Keep build dependencies updated
  • Test with different server configurations
  • Document custom build parameters

Integration {#integration}

CI/CD Pipelines

# Example GitHub Actions workflow
- name: Build Worker Binary
  run: |
    VERSION=${{ github.sha }} \
    SERVER_URL=https://api.bytebiota.com \
    python3 scripts/build_worker.py

Automated Deployment

# Build and deploy script
#!/bin/bash
VERSION=$(date +%Y.%m.%d)
SERVER_URL=https://prod.bytebiota.com

# Build
python3 scripts/build_worker.py

# Deploy
scp releases/v${VERSION}/*.tar.gz deploy@server:/opt/bytebiota/