Two spark applications listen on same port on same machine

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Two spark applications listen on same port on same machine

Moein Hosseini
I've submitted two spark applications in cluster of 3 standalone nodes in near the same time (I have bash script to submit them one after one without delay). But something goes wrong. In the master UI, Running applications section show both of my job with true configuration (cores, memory and different application-id) but both of redirect to port number 4040 which is listen by second submitted job.
I think it could be race condition in UI but found nothing in logs. Could you help me to investigate where should I look for reason?

Best Regards
Moein

--

Moein Hosseini
Data Engineer
mobile: <a href="tel:+98+912+468+1859" style="background-color:transparent;color:rgb(0,0,1)" target="_blank">+98 912 468 1859
site: www.moein.xyz
email: [hidden email]
linkedin
twitter

Reply | Threaded
Open this post in threaded view
|

Re: Two spark applications listen on same port on same machine

Sean Owen-2
Two drivers can't be listening on port 4040 at the same time -- on the same machine. The OS wouldn't allow it. Are they actually on different machines or somehow different interfaces? or are you saying the reported port is wrong?

On Wed, Mar 6, 2019 at 12:23 PM Moein Hosseini <[hidden email]> wrote:
I've submitted two spark applications in cluster of 3 standalone nodes in near the same time (I have bash script to submit them one after one without delay). But something goes wrong. In the master UI, Running applications section show both of my job with true configuration (cores, memory and different application-id) but both of redirect to port number 4040 which is listen by second submitted job.
I think it could be race condition in UI but found nothing in logs. Could you help me to investigate where should I look for reason?

Best Regards
Moein

--

Moein Hosseini
Data Engineer
mobile: <a href="tel:+98+912+468+1859" style="background-color:transparent;color:rgb(0,0,1)" target="_blank">+98 912 468 1859
site: www.moein.xyz
email: [hidden email]
linkedin
twitter

Reply | Threaded
Open this post in threaded view
|

Re: Two spark applications listen on same port on same machine

Moein Hosseini
I'm sure just the first one listen on port, but in master UI, both application redirects to same machie, same port. Just as I checked url, they redirects to application ui of first sumbitted one. So I think it could be only problem in UI.

On Wed, Mar 6, 2019, 10:29 PM Sean Owen <[hidden email]> wrote:
Two drivers can't be listening on port 4040 at the same time -- on the same machine. The OS wouldn't allow it. Are they actually on different machines or somehow different interfaces? or are you saying the reported port is wrong?

On Wed, Mar 6, 2019 at 12:23 PM Moein Hosseini <[hidden email]> wrote:
I've submitted two spark applications in cluster of 3 standalone nodes in near the same time (I have bash script to submit them one after one without delay). But something goes wrong. In the master UI, Running applications section show both of my job with true configuration (cores, memory and different application-id) but both of redirect to port number 4040 which is listen by second submitted job.
I think it could be race condition in UI but found nothing in logs. Could you help me to investigate where should I look for reason?

Best Regards
Moein

--

Moein Hosseini
Data Engineer
mobile: <a href="tel:+98+912+468+1859" style="background-color:transparent;color:rgb(0,0,1)" target="_blank" rel="noreferrer">+98 912 468 1859
site: www.moein.xyz
email: [hidden email]
linkedin
twitter