Release v0.2.17

Lightning AI ⚡ is excited to announce the release of LitServe v0.2.17

Highlights

Automatic Worker Restart

LitServe now supports automatic restarting of inference workers when they die, ensuring high availability and resilience in production environments. This prevents server shutdown due to isolated worker failures and maintains service continuity.

import litserve as ls

server = ls.LitServer(
    MyAPI(),
    restart_workers=True,  # Automatically restart failed workers
    workers_per_device=4
)
server.run()

When a worker terminates unexpectedly, the server automatically spawns a replacement, keeping requests flowing without interruption.

Changes

Added

• Add support for restarting the inference worker when they die by @tchaton in #624

Changed

• Update README to reflect inference engines terminology by @williamFalcon in #625
• chore: drop support for Python 3.9 by @bhimrazy in #641

Fixed

• Add warning for dict/set outputs in batched predict to catch edge cases by @Copilot in #612
• fix(sdk): Reduce the quantity of warning emitted by @tchaton in #631
• fix(litServe): Use asyncio.sleep instead of time.sleep by @tchaton in #633
• fix(cli): lightning-sdk installation process with uv by @bhimrazy in #640
• Fix on_request callback not triggering for API specs by @bhimrazy in #642

Chores

• Bump the gha-updates group with 2 updates by @dependabot[bot] in #626
• [pre-commit.ci] pre-commit suggestions by @pre-commit-ci[bot] in #629
• feat(litServe): Bump version 0.2.17 by @tchaton in #632
• Bump actions/checkout from 5 to 6 in the gha-updates group by @dependabot[bot] in #636
• [pre-commit.ci] pre-commit suggestions by @pre-commit-ci[bot] in #637
• Bump mypy from 1.18.2 to 1.19.0 by @dependabot[bot] in #639

🧑‍💻 Contributors

Thank you ❤️ to all contributors for making LitServe better!

Full Changelog: v0.2.16...v0.2.17

v0.2.16

What's Changed

• bump linting to min python version py3.9 by @Borda in #597
• enable testing with minimal requirements by @Borda in #596
• Fix duplicate/multiple middleware initialization by @geeksambhu in #601
• Support async LitAPI.health() and await it in /health by @KAVYANSHTYAGI in #604
• fix: Swagger UI message print when `disable_openapi_url=False by @bhimrazy in #620
• fix/req-middleware-duplication by @bhimrazy in #618

New Contributors

• @geeksambhu made their first contribution in #601
• @KAVYANSHTYAGI made their first contribution in #604
• @Abdul-0x4A made their first contribution in #621
• @dmitsf made their first contribution in #623

Full Changelog: v0.2.15...v0.2.16

v0.2.15

What's Changed

• ci: add testing cron by @Borda in #585
• fix(ci): handle sentinel input in request_queue gracefully by @aniketmaurya in #589
• pytest uses just one config by @Borda in #586
• handle invalid operation in zmq transport by @emmanuel-ferdman in #591
• feat(litserve): Add support for loading TLS certificates for the user by @tchaton in #592
• Release 0.2.15 by @aniketmaurya in #593

New Contributors

• @emmanuel-ferdman made their first contribution in #591
• @tchaton made their first contribution in #592

Full Changelog: v0.2.14...v0.2.15

v0.2.14

What's Changed

• Bump mypy from 1.16.0 to 1.16.1 by @dependabot[bot] in #565
• support mcp package less than v1.10.0 by @rongfengliang in #567
• fix: OpenAIEmbeddingSpec setup check for multi endpoint by @rongfengliang in #568
• [pre-commit.ci] pre-commit suggestions by @pre-commit-ci[bot] in #571
• Fix: pre-commit errors on main by @bhimrazy in #574
• refactor: validation logic to pre_setup method in Embed Spec, where there is access to correct api instance by @bhimrazy in #573
• uv for CI - faster CI by @aniketmaurya in #563
• feat: openapi url by @lorenzomassimiani in #578
• Feat/override-spec-api-path by @bhimrazy in #577
• Pre-release 0.2.14a0 by @aniketmaurya in #579
• migrate arguments from LitServe to LitAPI in tests by @aniketmaurya in #576
• chore: Add warnings for custom API paths in OpenAI Chat and Embed specs by @bhimrazy in #581
• 🚀 Feature: warning for heavy init method in LitAPI by @SN4KEBYTE in #582
• Release 0.2.14 by @aniketmaurya in #583

New Contributors

• @rongfengliang made their first contribution in #567
• @SN4KEBYTE made their first contribution in #582

Full Changelog: v0.2.13...v0.2.14

v0.2.13

What's Changed

• Add mcp support in README by @aniketmaurya in #540
• Add Dependabot for Pip & GitHub Actions by @Borda in #541
• Comprehensive docstrings by @aniketmaurya in #545
• Update numpy requirement from <2.0 to <3.0 by @dependabot in #543
• Bump mypy from 1.11.2 to 1.16.0 by @dependabot in #544
• Bump the gha-updates group with 2 updates by @dependabot in #542
• Add reasoning effort parameter to OpenAI Spec ChatCompletionRequest by @bhimrazy in #548
• Fix async streaming with OpenAISpec by @aniketmaurya in #552
• add test for async-sync function invocation handler by @aniketmaurya in #553
• add pytest marker for unit, integration and e2e tests by @aniketmaurya in #554
• Update LitServer initialization parameters for type safety by @aniketmaurya in #555
• Send blocking CPU operations to thread for async conversion. by @aniketmaurya in #556
• Release 0.2.13rc1 by @aniketmaurya in #557
• return unmodified request for OpenAI chatcompletion decode_request by @aniketmaurya in #558
• MCP package dependency check by @aniketmaurya in #561
• Improve error handling and logging for streaming by @aniketmaurya in #562
• Release 0.2.13 by @aniketmaurya in #564

Full Changelog: v0.2.12...v0.2.13

v0.2.13rc1

What's Changed

• Add mcp support in README by @aniketmaurya in #540
• Add Dependabot for Pip & GitHub Actions by @Borda in #541
• Comprehensive docstrings by @aniketmaurya in #545
• Update numpy requirement from <2.0 to <3.0 by @dependabot in #543
• Bump mypy from 1.11.2 to 1.16.0 by @dependabot in #544
• Bump the gha-updates group with 2 updates by @dependabot in #542
• Add reasoning effort parameter to OpenAI Spec ChatCompletionRequest by @bhimrazy in #548
• Fix async streaming with OpenAISpec by @aniketmaurya in #552
• add test for async-sync function invocation handler by @aniketmaurya in #553
• add pytest marker for unit, integration and e2e tests by @aniketmaurya in #554
• Update LitServer initialization parameters for type safety by @aniketmaurya in #555
• Send blocking CPU operations to thread for async conversion. by @aniketmaurya in #556
• Release 0.2.13rc1 by @aniketmaurya in #557

Full Changelog: v0.2.12...v0.2.13rc1

v0.2.12

What's Changed

• Docs: Address text-davinci-003 deprecation and new API structure in NewsAgent example by @kumarrah2002 in #521
• Add tests for async streaming loops by @bhimrazy in #522
• [pre-commit.ci] pre-commit suggestions by @pre-commit-ci in #523
• ci: Increase CI test timeout to 15 minutes by @bhimrazy in #526
• Enhance process and thread naming in LitServer by @aniketmaurya in #528
• Fix health and info endpoints when multiple LitAPIs are specified by @vrdn-23 in #529
• Pre-release 0.2.12.dev0 by @aniketmaurya in #531
• created shutdown endpoint with API key security and custom passed tests by @kumarrah2002 in #525
• chore: Update CODEOWNERS by @andyland in #533
• Shutdown server when workers crash by @andyland in #532
• Improve perf test connection pool by @aniketmaurya in #537
• make dependency installation check as utility function by @aniketmaurya in #535
• input schema extraction for MCP server by @aniketmaurya in #536
• Enable MCP server by @aniketmaurya in #534
• Release 0.2.12 by @aniketmaurya in #539

Full Changelog: 0.2.11...v0.2.12

v0.2.12.dev0

What's Changed

• Docs: Address text-davinci-003 deprecation and new API structure in NewsAgent example by @kumarrah2002 in #521
• Add tests for async streaming loops by @bhimrazy in #522
• [pre-commit.ci] pre-commit suggestions by @pre-commit-ci in #523
• ci: Increase CI test timeout to 15 minutes by @bhimrazy in #526
• Enhance process and thread naming in LitServer by @aniketmaurya in #528
• Fix health and info endpoints when multiple LitAPIs are specified by @vrdn-23 in #529
• Pre-release 0.2.12.dev0 by @aniketmaurya in #531
• created shutdown endpoint with API key security and custom passed tests by @kumarrah2002 in #525

Full Changelog: 0.2.11...v0.2.12.dev0

v0.2.11

What's Changed

• Remove un used imports and use enum by @mo7amed-3bdalla7 in #493
• moving max_batch_size in README to inside SimpleLitAPI() by @kumarrah2002 in #495
• rename deploy command by @aniketmaurya in #497
• missing fstring typo by @mathematicalmichael in #498
• fix: OpenAI Spec validations to work with async LitAPI by @bhimrazy in #499
• chore: Update CODEOWNERS to include additional reviewers by @aniketmaurya in #503
• fix OpenAI embedding spec for batching by @aniketmaurya in #500
• Release 0.2.11a0 by @aniketmaurya in #504
• add shutdown endpoint w/ test by @kumarrah2002 in #507
• Revert "add shutdown endpoint w/ test" by @aniketmaurya in #509
• add fpdb for multiprocess debugging using pdb by @aniketmaurya in #508
• Enhance logging configuration to support optional Rich logging by @aniketmaurya in #510
• Release 0.2.11a1 by @aniketmaurya in #511
• move stream, endpoint path, loop to LitAPI initialization by @aniketmaurya in #512
• Support multiple LitAPIs for inference process and endpoints by @aniketmaurya in #513
• Release 0.2.11a2 by @aniketmaurya in #516
• remove decode and encode methods from README by @aniketmaurya in #515
• decouple request handler and add test by @aniketmaurya in #517
• Support stream with non-stream LitAPIs by @aniketmaurya in #518
• Improve developer experience by @aniketmaurya in #519

New Contributors

• @mo7amed-3bdalla7 made their first contribution in #493
• @kumarrah2002 made their first contribution in #495
• @mathematicalmichael made their first contribution in #498

Full Changelog: v0.2.10...0.2.11

v0.2.11.a2

What's Changed

• move stream, endpoint path, loop to LitAPI initialization by @aniketmaurya in #512
• Support multiple LitAPIs for inference process and endpoints by @aniketmaurya in #513
• Release 0.2.11a2 by @aniketmaurya in #516
• remove decode and encode methods from README by @aniketmaurya in #515

Full Changelog: v0.2.11a1...v0.2.11.a2