Scaling Databricks Apps: A Practical Guide for Developers

Congratulations on successfully building an application on Databricks Apps! As your user base expands from a handful to thousands, the demands for quick response times and high uptime increase. At the same time, managing costs can become a concern as you may face hefty charges for larger compute instances.

This guide will explore how to efficiently scale Databricks Apps through four key phases: design, build, test and deploy, and operate. Our primary focus is on enhancing application performance and resilience at scale, while we won’t delve into deployment automation or performance tuning of other Databricks services integrated with the application.

Establish Clear Performance Goals

The journey to scalable architecture begins even before writing any code. It's crucial to define user requirements and determine where the application will operate and where its data will reside.

Identify two critical targets:

• Performance: Define latency expectations for key actions, ideally using a percentile metric such as p95 to capture outlier experiences not reflected in average metrics. Don't forget to account for UI rendering time and those cold starts. For streaming applications, consider metrics like time to first token.
• Scalability: Evaluate peak user loads, frequency of usage, and anticipated growth. Be mindful of busy periods, such as month-end spikes, as these are factors that your application must accommodate within its defined latency and error-rate limits.

Additionally, consider which functionalities need to persist through restarts and ensure that essential dependencies can support this requirement. This assessment will inform your state management and instance scaling strategies.

Designing the Architecture for Scalability

When designing the architecture, it's vital to allocate app compute resources efficiently. The purpose of app compute is to handle UI serving and coordinate service calls, rather than doing heavy processing.

Instead of keeping long-running requests active, consider running background jobs. The UI can periodically check for updates rather than holding open requests, which ensures responsiveness.

If your app must execute code written by users or generated through an agent, leverage Databricks Sandbox, an isolated environment that helps manage CPU and memory usage away from user-serving instances.

Choosing the Right Framework

As user consumption grows, your app can be scaled in two directions: vertical scaling (moving to a larger compute instance with more resources) or horizontal scaling (adding more instances to distribute the load). Know that heavy computations usually dictate scaling limits. In many situations, horizontal scaling helps manage user requests more effectively.

For new applications, we recommend using AppKit, a TypeScript SDK that provides built-in caching, telemetry, and error processing, streamlining the development process. Databricks Apps supports building applications in both Python and Node.js, allowing for flexibility based on your needs.

Handling Application State

A significant part of successfully scaling your app is managing application state. Identify what state your application holds and which components must survive restarts. Classify your data into soft and hard state:

• AI applications: Keep context and outputs in soft state while ensuring important checkpoints are saved.
• Workflow applications: Poll or stream status updates to the UI, while saved steps represent hard state.
• Dashboards: Retain filter values and preferences in memory while preserving user-specific configurations as hard state.

Databricks Apps makes an effort to maintain session affinity, which typically retains per-user caches even during restarts. Thus, it’s important for your architecture to allow state reconstruction from hard state to ensure uninterrupted service.

Code Considerations for Scalability

With the architecture in place, it’s time to write code optimized for load handling and state persistence. Implementing telemetry early in the development process is crucial, as it provides critical data points for future performance tuning.

Utilize asynchronous programming to minimize wasted processing time. Many applications find themselves waiting on external database queries or API calls, so non-blocking methods in such cases allow other requests to proceed while waiting.

Efficiently managing caching is also important. Ensure caches don't inadvertently expose user data by controlling cache keys based on authenticated identities. For shared resources, let single requests refresh expired cache entries to limit the load on your data warehouse.

Testing and Final Deployment

Conduct a stress test before going live. Utilize a tool such as Locust to simulate real user behavior at peak loads, collecting metrics on request rates, latency, and failure rates. Visual metrics will identify which aspect of your instance, memory, or CPU triggers scalability limits.

During the load test, examine how well the app recovers from restarts and whether any user sessions were compromised. This information will guide adjustments to ensure reliable functionality under load.

Finally, remember that graceful shutdown processes are essential to maintain user experience during redeployments or updates. Timing is key, allowing ongoing requests to wrap up properly before moving to a new instance.

By carefully considering these aspects of scaling your Databricks Apps, you can ensure a robust and responsive application that meets user demands while effectively managing resources.