SageWorks CloudWatch
Need Help?
The SuperCowPowers team is happy to give any assistance needed when setting up AWS and SageWorks. So please contact us at sageworks@supercowpowers.com or on chat us up on Discord
The SageWorks framework continues to 'flex' to support different real world use cases when operating a set of production machine learning pipelines. As part of this we're including CloudWatch log forwarding for any use of the SageWorks API (Dashboard, Glue, Lambda, Notebook, Laptop, etc).
Functionality
The SageWorks logging setup includes the addition of a CloudWatch 'Handler' that forwards all log messages to the SageWorksLogGroup
Individual Streams
Each process running SageWorks will get a unique individual stream.
- dashboard/* (any logs from Web Dashboard)
- glue/* (logs from Glue jobs)
- lambda/* (logs from Lambda jobs)
- docker/* (logs from Docker containers)
- laptop/* (logs from laptop/notebooks)
Since many jobs are run nightly/often, the stream will also have a date on the end... glue/my_job/2024_08_01_17_15
AWS CloudWatch made Easy
Logs in Easy Mode
The SageWorks cloud_watch
command line tool gives you access to important logs without the hassle. Automatic display of important event and the context around those events.
The cloud_watch
script will automatically show the interesting (WARNING and CRITICAL) messages from any source within the last hour. There are lots of options to the script, just use --help
to see options and descriptions.
Here are some example options:
# Show important logs in last 12 hours
cloud_watch --start-time 720
# Show a particular stream
cloud_watch --stream glue/my_job
# Show/search for a message substring
cloud_watch --search SHAP
# Show ALL logs (includes debug)
cloud_watch --search ALL
# Show messages this log 'level' or above
cloud_watch --search IMPORTANT
cloud_watch --search WARNING
cloud_watch --search ERROR
cloud_watch --search CRITICAL
More Information
Check out our presentation on SageWorks CloudWatch
Questions?
The SuperCowPowers team is happy to anser any questions you may have about AWS and SageWorks. Please contact us at sageworks@supercowpowers.com or on chat us up on Discord