ByteBiota worker distribution roadmap

Back to Home

ByteBiota worker distribution roadmap

Overview

This page tracks how we will ship the distributed worker as a standalone artifact. We now treat the work as two coordinated streams:

  1. Phase A – Worker artifact packaging (current focus). Produce reproducible executables for Windows, Linux, and macOS without changing worker behaviour.
  2. Phase B – Worker self-update (follow-up plan). Layer new services and CLI affordances so workers can download and install updates on their own. This work is captured in worker-self-update.md.

Splitting the roadmap lets us ship packaging improvements without waiting on the larger protocol and infrastructure work required for self-updates.

Current state snapshot

  • The worker entry point lives in src/bytebiota/worker/worker.py and exposes main() for CLI execution.
  • python -m bytebiota.worker.worker is currently launched by deployment scripts; PyInstaller spec and build script have been implemented.
  • The server handshake still returns the fixed string "1.0.0" and no API advertises download locations or required versions.
  • GitHub Actions runs deploy.yml only for release tags (release/server-*, server-v*) or manual dispatches, and the worker packaging workflow (build-worker.yml) which emits tagged release artifacts.

Phase A has been completed with local build capabilities. The phased roadmap below aligns with what is actually in the repository today.

Phase A – Worker artifact packaging {#phase-a-packaging}

Goal: deliver signed, versioned worker binaries from CI so operators can install them manually.

Deliverables

  1. PyInstaller specification. Add worker.spec under the repository root.
  2. Entry script: worker_entry.py (simple wrapper that imports and calls worker main function).
  3. pathex=['src'] so package imports resolve.
  4. Minimal hidden imports limited to modules that actually ship today (for example bytebiota.worker.resource_limiter, bytebiota.worker.connection_manager, bytebiota.run_logger). Do not reference future modules such as bytebiota.worker.updater.
  5. Collect data files with collect_data_files('bytebiota', excludes=['tests']) to keep configuration defaults bundled without dragging in test fixtures.
  6. Cross-platform build helper. Introduce scripts/build_worker.py (invoked via thin OS-specific wrappers) to standardise local builds.
  7. Generate date-based versions (YYYY.MM.DD[.N]) using the system clock and optional $VERSION override.
  8. Normalise platform (linux, macos, windows) and architecture (amd64, arm64).
  9. Move the PyInstaller output to dist/bytebiota-worker-${VERSION}-${PLATFORM}-${ARCH} and gzip on Unix platforms.
  10. Emit SHA-256 checksum files alongside the artifacts and create platform-appropriate archives.
  11. GitHub Actions workflow. .github/workflows/build-worker.yml runs on tag pushes and manual dispatch.
  12. Matrix over ubuntu-latest, macos-latest, and windows-latest.
  13. Uses consistent bash steps to share the Python orchestrator across platforms.
  14. Caches pip downloads to reduce runtimes.
  15. Uploads archives and dist outputs using the GitHub upload action with consistent naming.
  16. Documentation and release notes. Update DEPLOYMENT.md (manual install steps) once artifacts exist, and document the workflow triggers in wiki/operations/deployment/deployment-guide.md.

Work breakdown

Sequence Task Key files Acceptance criteria
A1 Scaffold worker.spec worker.spec Running pyinstaller worker.spec on Linux produces a binary that starts with --help and includes all runtime dependencies.
A2 Author scripts/build_worker.py scripts/build_worker.py Local run builds platform/arch directories with checksums; rerunning with the same version overwrites outputs deterministically.
A3 Embed version metadata src/bytebiota/worker/worker.py, worker.spec Worker CLI prints the packaged version via --version, using a value injected from the build script.
A4 Publish GitHub workflow .github/workflows/build-worker.yml Manual workflow dispatch produces artifacts for all matrix entries and uploads checksums + manifest.
A5 Update operator docs DEPLOYMENT.md, wiki/operations/deployment/deployment-guide.md Docs reference the new artifacts, include manual install steps, and link to workflow outputs.

Implementation checklist

  • [ ] Add smoke test target make package-worker (optional convenience) to invoke scripts/build_worker.py locally.
  • [ ] Capture build logs and attach to GitHub workflow artifacts for debugging.
  • [ ] Register artifact retention policy (14 days default, override if release cadence slower).
  • [ ] Validate binaries on each OS by launching bytebiota-worker --help (Windows .exe run via pwsh Start-Process -Wait).
  • [ ] Feed checksum outputs into release manifest (used in Phase B).
  • [ ] Local dry run: execute python3 scripts/build_worker.py on Linux/macOS (or py -3 scripts\build_worker.py on Windows) and confirm the binary launches with ./dist/.../bytebiota-worker --help (or bytebiota-worker.exe --help).

Dependencies and coordination

  • Requires existing Python build toolchain (PyInstaller, poetry export). Add new entries to requirements-dev.txt if missing.
  • Coordinate with security to obtain signing certificates before publishing if we want authentic code signatures; otherwise note that signing is deferred.
  • Align version numbering with server expectations (YYYY.MM.DD.N becomes worker_version environment variable for deployments).

Validation checklist

  • Local dry run: execute python3 scripts/build_worker.py on Linux/macOS (or py -3 scripts\build_worker.py on Windows) and confirm the binary launches with ./dist/.../bytebiota-worker --help (or bytebiota-worker.exe --help).
  • CI smoke test: extend the workflow to run bytebiota-worker --version after packaging using pyinstaller --version-file metadata or by invoking the CLI with --version once added.
  • Manual verification: compare checksum outputs with locally generated hashes before publishing.

Out-of-scope (defer to Phase B)

  • Any REST endpoints related to update discovery (update_api_service).
  • Worker CLI flags such as --force-update or environment-driven auto-upgrade toggles.
  • Platform-specific installers (MSI, PKG) and signature automation.

Phase B – Worker self-update (future)

Self-update requires new modules (worker updater, server API, version negotiation) and changes to the handshake contract. The detailed plan now lives in worker-self-update.md. That document depends on the artifact naming conventions defined above.