Session 18 monitering and logging - Azure Monitor , Log analytics , and job notification

 After developing the code, we deploy it into the production environment.

To monitor and logging the jobs run in the real time systems in azure 

we have scheduled the jobs under the workflow , we haven't created any monitoring or any matrics .

After a few times, the job failed, but we don't know because we haven't set up any monitoring, and every time we can't navigate to workspace-> workflows, under runs to see to check whether the job has been successfully running or not and in real time there will be nearly 100 jobs or more jobs to run 

In real time, the production support team will monitor the process.

Under the workflow, there is an option called Job notification. After setting the job notification, we can set a notification to email .

if we click the date and time its takes us to the notebook which is scheduled there we can able to see the error where it happens .

order to see more details, we need to under Spark tab, where we have the option to view logs ( that is driver logs ) .

Under each table that we created or any resource, there is a permission tab where we can see who is all able to access the particular resources.