Logs and monitoring

Job logs

For all data generation jobs, the job logs provide detailed information about the job processing. Syntho support might request the job logs to help diagnose issues.

Download job logs

Go to your workspace, select Jobs > Job history > Job summary and select the Download logs button to download the logs of a data generation job.

Click on "Jobs" to go to the Job history panel and Select "Download logs" to download the job logs

Logs redaction

To avoid leaking values, by default Syntho prevents including sensitive values or redacts sensitive values (e.g. passwords) from the logs.

Monitor cluster utilization in Ray dashboard

Syntho uses Ray to scale workloads.

Go to your workspace, select Jobs > Job history > Job summary and select the Ray dashboard button to view the Ray cluster utilization dashboard.

Ray dashboard cluster utilization

Syntho utilizes Ray for executing long-running tasks such as synthesizing, PII scans, and FK scans. To monitor these tasks, the Ray dashboard provides a real-time overview of job execution, system resource usage, and logs.

Ray dashboard overview

By leveraging the Ray dashboard, you can efficiently track job execution, detect potential bottlenecks, and access relevant logs to troubleshoot issues.

Jobs

Jobs tab displays all running and past jobs. A constant background job from the Core API ensures system health. You can access job-specific logs, flame graphs, and execution details via the Actions column.

Jobs tab

Cluster

Cluster tab provides a summary of all active Ray nodes, including key resource metrics like CPU, memory, and disk usage. You can click on a node ID to reveal a detailed performance breakdown.

Cluster tab

Actors

Actors tab lists active and dead actors. While generally not needed, you can inspect memory consumption per actor if required.

Actors tab

Monitoring specific ray jobs

For real-time tracking of synthesizing, PII scans, or FK scans:

1. Start the job and open the Ray dashboard.

2. Use the Cluster tab to monitor resource consumption.

3. Upon completion, check the Jobs tab for job status updates (success or failure).

Note that Syntho does not use Ray Serve, so this section in the dashboard is not applicable.