Tag: Data Engineering
Setting Up My Spark Lab
When learning Spark and testing with small datasets, I can simply run a local Spark instance with the following command which essentially creates a local Spark instance using all cores. This local instance has no worker and the driver will handle all jobs and tasks. However, it is more interesting and useful to run a…