Jupyter connect to spark driver in kubernetes.

Submitted 1 month, 1 week ago

Ticket #447

Views 41

Language/Framework Spark

Priority Low

Status In-Progress

I want to connect jupyter notebook to spark driver and worker node and trying to access Minio S3 bucket in kubernetes. But I am able to launch jupyter notebook and spark driver and worker node. But while I try to run my code using spark driver it not work but using local spark i am able to access. Please help me i wanna run my code using spark driver and worker node in kubernetes.

Ph :- 8077044733

Sonu

Submitted on Aug 29, 24

sonu

can you share your code for me to debug on this? - Vengat 1 month, 1 week ago

Hi Sir I have kubernetes installed i have 4 raspberry pi each contain 8Gb Ram = 32 and core 16 Core. You can take remote desktop my raspberry pi using Connect.raspberrypi.com i will give you login and password. My mail id sonukr023@gmail.com I am trying this last one month I'm not successful. But same i can able to do in docker compose file. Please help me My ph : +91 8077044733 - sonu 1 month, 1 week ago

I just need your error which you are getting . PLease share that - Vengat 1 month, 1 week ago

Hi Sir i just resolved the issue. Thank you so much. My cluster was not access the different node so I mentioned the ip address to access the node where my spark driver was running. Below config was missing now I'm able to access my spark driver node using jupyter notebook. Thank you so much 🙏 config("spark.driver.host", "10.42.0.68")\ .config("spark.driver.bindAddress", "10.42.0.68")\ - sonu 1 month ago

add a comment

Latest Blogs

Step by Step guide to deploy Django Web App on Heroku

Python XML - Read/Write and Create

Day 6-Mastering PySpark: A 100-Day Challenge

How to Implement the Django filters

Different way of processing the GET request in Javascript

Django Heroku Deployment Steps

How to setup the Notification in website using Javascript and Django

How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 1

How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 2

Python Enumerate in Django Choice Field

Creating A Popup Modal Login Form Using HTML5 And CSS and Bootstrap.

Simple Search App using Django API and Vanilla Javascript

Automation Testing Life Cycle

How to Integrate Django and Vue JS Part -1

Steps to deploy Django App into AWS

Building Live streaming web app with FastAPI and ReactJS

SOME STEPS TO FOLLOW BEFORE STARTING A PROJECT :

What is the secret behind SUCCESS?

Success

Django Static Files in Local

Creation of web pages for register,login ,logout (PART-2)

Data Analytics in a Nutshell

Python’s Generator and Yield

What is Mean? Why is it required?

IMPLEMENTATION OF SEARCH BAR USING DJANGO (in any website)

Creation of webpages for register ,login ,logout (PART-1)

Web designer or Front End Developer? Key Differences

Title: Unraveling the Wonders of Quantum Computing: A Leap into the Future

Python Desktop APP to Upload Image into AWS S3 with Progress Bar

Building a Gallery App with FastAPI and React in AWS S3

Tips for Optimizing Apache Spark Queries — Part 1 (Data Partitioning)

Building a Real-time Feedback System with Django, Kafka, Spark, and S3

Ongoing Knowledge - Keep Learning ang Keep Sharing

Custom Model Managers In Django

Python Merge PDF files

Python Challenge 1 - Capital indexes

How to Integrate Django and Vue JS Part -2( Creating Simple Todo App)

Program vs Process vs Thread

Python Multiprocessing with Real Time Example

Unlocking the Power of Caching with Python, Relational Database & Redis Unleashed

Stay Hydrated with a Python Script

Day 8 UDF -Mastering PySpark: A 100-Day Challenge

Resilient in Spark— My Understanding

Pyspark Row : asDict(): A 100-Day PySpark Challenge

PySpark — Dataframe Operations CheatSheet

Snowflake Database Zero to Hero - Let's Learn Together