Amazon Simple Storage Service (S3)

Source and Destination Databases

Destination only

This connector can only be used as a destination for writing your generated data.

• Supported File Types: Parquet and ORC

• Supported Partitioning: Horizontal partitioning based on the write batch size (i.e. each batch will be written to a separate file). Please also give an example of file output structure.

Before you begin

Before you begin, gather this connection information:

• Get the connection details to connect with your S3 bucket

File formats

Supported file type formats include:

• Parquet

• ORC

Output format

Syntho's S3 output connector will write all generated data to files as follows:

• Each generated table will be written to a Parquet file in the following format: {schema-name}-{table_name}_part_{part_number}.parquet

• The number of rows in a single Parquet file (part) is defined by the batch_generate size. All the Parquet parts of a single table will be stored in their own directory, which is dedicated to that particular table.

• Each folder name will use the following format:

  {schema_name}.{table_name}

Connect and set up the workspace

Launch Syntho and select Connect to a database, or under Create workspace > Destination Database, select S3. For a complete list of data connections, select More under From database. Then do the following:

1. Enter the bucket name.

2. Enter the region name.

3. Enter the port number.

4. Enter the AWS access key id.

5. Enter the AWS secret access key.

6. Enter the prefix. If Syntho can't make the connection, verify that your credentials are correct. If you still can't connect, your computer is having trouble locating the server. Contact your network administrator or database administrator.

Supported data types

• The supported data types for ORC files are specified in the Apache Arrow documentation.

Logical type	Mapped Arrow type
BOOLEAN	Boolean
BYTE	Int8
SHORT	Int16
INT	Int32
LONG	Int64
FLOAT	Float32
DOUBLE	Float64
STRING	String/LargeString
BINARY	Binary/LargeBinary/FixedSizeBinary
TIMESTAMP	Timestamp/Date64
TIMESTAMP_INSTANT	Timestamp
LIST	List/LargeList/FixedSizeList
MAP	Map
STRUCT	Struct
UNION	SparseUnion/DenseUnion
DECIMAL	Decimal128/Decimal256
DATE	Date32
VARCHAR	String
CHAR	String

Errors can occur during data conversion when writing to ORC files if unsupported data types are involved.

Limitations & considerations

Contact your Syntho contact person to discuss possible limitations regarding this connector.

• For ORC files, columns full of None values which are of type Char, String or Varchar will be written as "None" (i.e. a string value) to the destination database instead of None.