View and adjust generation settings

The final step in the data generation wizard allows you to adjust the Generation settings.

Advanced generation settings

The following advanced settings can be adjusted for your generation job:

• Read batch size: The maximum number of rows per batch to read from each table in the source database. The allowed value is integer (number of rows). The default batch size is 100k. You can increase the value to improve reading speeds, at the costs of a larger memory usage, and vice versa.

• Write batch size: The maximum number of rows per batch to insert in each table in the destination database. The allowed value is integer (number of rows). The default batch size is 100k. You can increase the value to improve writing speeds, at the costs of a larger memory usage, and vice versa.

• Maximum number of connections: The maximum number of connections that can be made to the database during the writing. A higher number can speed up the process as it allows for more parallel operations. However, if you are reading from a production database, this might not be desirable.

Known Issues

Parameter Limit Error When data is inserted into the destination database you may encounter an error indicating that the SQL query parameter limit has been exceeded. This occurs because large write batch sizes can exceed the driver’s maximum allowable parameters for a single query. For instance, you might see an error like:

The SQL contains 16970 parameter markers, but 1000010 parameters were supplied'

If you experience this error, reducing the "Write batch size" is recommended to stay within the driver’s parameter limits. As a starting point, you can reduce the batch size from 1 million datapoints (1M) to 500k datapoints (0.5M), half the write batch size again even lower, if the errors persist. The exact limit can vary per environment.