9. Large workloads
Large workloads
Speeding up data generation jobs and reducing memory footprint
Working with large databases can significantly impact the performance and success of your synthetic data generation jobs. These tips will help you configure your workspace for large workloads by minimizing memory consumption and optimizing execution speed.
Lowering memory footprint
To reduce memory usage and avoid potential timeouts or job failures, consider these strategies:
Decrease parallel connections: Lower the number of concurrent connections to reduce memory usage.
Decrease read and write batch size: Smaller batches consume less memory per operation.
Limit free text PII detection: This is a resource-intensive process. Only enable it when absolutely necessary.
Reduce the number of training rows (AI synthesis only): Limiting the training data size speeds up processing and conserves resources.
Speeding up data generation jobs
To accelerate data generation for large-scale datasets, apply the following optimizations:
Increase parallel connections: More connections can speed up data reading and writing through parallel execution.
Enable schema-independent scheduling: By removing constraints in the destination schema, Syntho can parallelize processing based on the number of records instead of schema dependencies.
Write to Parquet instead of a database: Writing directly to a database is often slower. When dealing with very large datasets, consider exporting to efficient columnar file formats like Parquet.
Interactive guide: How to handle large workloads
Follow the interactive guide below to handle large workloads
Best practice
Always aim to use the minimal viable dataset to validate your configurations before executing large jobs. Scaling up becomes much easier and more stable when you're confident in your setup.
Last updated
Was this helpful?