> For the complete documentation index, see [llms.txt](https://docs.syntho.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.syntho.ai/overview/get-started/syntho-bootcamp/9.-large-workloads.md). # 9. Large workloads​ ## Large workloads ### Speeding up data generation jobs and reducing memory footprint Working with large databases can significantly impact the performance and success of your synthetic data generation jobs. These tips will help you configure your workspace for large workloads by minimizing memory consumption and optimizing execution speed. *** ### Lowering memory footprint To reduce memory usage and avoid potential timeouts or job failures, consider these strategies: * [**Decrease parallel connections**](/configure-a-data-generation-job/generation-and-validation/view-and-adjust-generation-settings.md#advanced-generation-settings): Lower the number of concurrent connections to reduce memory usage. * [**Decrease batch size**](/configure-a-data-generation-job/generation-and-validation/view-and-adjust-generation-settings.md#advanced-generation-settings): Smaller batches consume less memory per operation. * [**Limit free text PII detection**](/configure-a-data-generation-job/configure-column-settings/duplicate/automatic-pii-discovery-and-de-identification-in-free-text-columns.md#use-syntho-pii-text-scanner): This is a resource-intensive process. Only enable it when absolutely necessary. * [**Reduce the number of training rows**](/configure-a-data-generation-job/configure-column-settings/ai-powered-generation.md#advanced-generator-settings) *(AI synthesis only)*: Limiting the training data size speeds up processing and conserves resources. *** ### Speeding up data generation jobs To accelerate data generation for large-scale datasets, apply the following optimizations: * [**Increase parallel connections**](/configure-a-data-generation-job/generation-and-validation/view-and-adjust-generation-settings.md#advanced-generation-settings): More connections can speed up data reading and writing through parallel execution. * **Enable schema-independent scheduling**: By removing constraints in the destination schema, Syntho can parallelize processing based on the number of records instead of schema dependencies. * **Write to Parquet instead of a database**: Writing directly to a database is often slower. When dealing with very large datasets, consider exporting to efficient columnar file formats like Parquet. ### Interactive guide: How to handle large workloads Follow the interactive guide below to handle large workloads {% embed url="" %} *** ### Best practice Always aim to use the minimal viable dataset to validate your configurations before executing large jobs. Scaling up becomes much easier and more stable when you're confident in your setup. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.syntho.ai/overview/get-started/syntho-bootcamp/9.-large-workloads.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.