Azure Data Lake Storage (ADLS)

Source and Destination Databases

Destination only

This connector can only be used as a destination for writing your generated data.

• Supported File Types: Parquet and ORC

• Supported Partitioning: Horizontal partitioning based on the write batch size (i.e. each batch will be written to a separate file). Please also give an example of file output structure.

Before you begin

Before you begin, review the items in the following list:

• Get the URL for your Azure storage endpoint.

• Depending on how you want to connect, prepare either:

  • the storage account name and storage account key, or

  • the connection string.

• Provide read/write permissions to the storage container that is used (i.e., the storage container name).

• Get the remote path, which is the relative path to the storage container that is used.

File formats

Supported file type formats include:

• Parquet

• ORC

Output format

Syntho's ADLS output connector will write all generated data to Parquet files as follows:

• Each generated table will be written to a Parquet file in the following format: {schema-name}-{table_name}_part_{part_number}.parquet

• The number of rows in a single Parquet file (part) is defined by the batch_generate size. All the Parquet parts of a single table will be stored in their own directory, which is dedicated to that particular table.

• Each folder name will use the following format:

  {schema_name}.{table_name}

Connect and set up the workspace

Launch Syntho and select Connect to a database, or under Create workspace > Destination Database, select ADLS. For a complete list of data connections, select More under From database. Then do the following:

1. Enter the remote path.

2. Enter the storage container name.

3. Either:

   • Enter the storage account name and the storage account key, or:

   • Enter the storage connection string.

4. Select Create Workspace. If Syntho can't make the connection, verify that your credentials are correct. If you still can't connect, your computer is having trouble locating the server. Contact your network administrator or database administrator.

Supported Data Types

• The supported data types for ORC files are specified in the Apache Arrow documentation.

Logical type	Mapped Arrow type
BOOLEAN	Boolean
BYTE	Int8
SHORT	Int16
INT	Int32
LONG	Int64
FLOAT	Float32
DOUBLE	Float64
STRING	String/LargeString
BINARY	Binary/LargeBinary/FixedSizeBinary
TIMESTAMP	Timestamp/Date64
TIMESTAMP_INSTANT	Timestamp
LIST	List/LargeList/FixedSizeList
MAP	Map
STRUCT	Struct
UNION	SparseUnion/DenseUnion
DECIMAL	Decimal128/Decimal256
DATE	Date32
VARCHAR	String
CHAR	String

Errors can occur during data conversion when writing to ORC files if unsupported data types are involved.

Limitations & Considerations

Contact your Syntho contact person to discuss possible limitations regarding this connector.

• For ORC files, columns full of None values which are of type Char, String or Varchar will be written as "None" (i.e. a string value) to the destination database instead of None.