9. Large workloads
Last updated
Was this helpful?
Working with large databases can significantly impact the performance and success of your synthetic data generation jobs. These tips will help you configure your workspace for large workloads by minimizing memory consumption and optimizing execution speed.
To reduce memory usage and avoid potential timeouts or job failures, consider these strategies:
Decrease parallel connections: Lower the number of concurrent connections to reduce memory usage.
Decrease batch size: Smaller batches consume less memory per operation.
Limit free text PII detection: This is a resource-intensive process. Only enable it when absolutely necessary.
Reduce the number of training rows (AI synthesis only): Limiting the training data size speeds up processing and conserves resources.
To accelerate data generation for large-scale datasets, apply the following optimizations:
Increase parallel connections: More connections can speed up data reading and writing through parallel execution.
Enable schema-independent scheduling: By removing constraints in the destination schema, Syntho can parallelize processing based on the number of records instead of schema dependencies.
Write to Parquet instead of a database: Writing directly to a database is often slower. When dealing with very large datasets, consider exporting to efficient columnar file formats like Parquet.
Follow the interactive guide below to handle large workloads
Always aim to use the minimal viable dataset to validate your configurations before executing large jobs. Scaling up becomes much easier and more stable when you're confident in your setup.
Last updated
Was this helpful?
Was this helpful?