Python Development

I'm trying the mongodb hadoop integration with spark but can't figure out how to make the j… Read more Add Jar To Pyspark When Using Notebook

I asked the question a while back for python, but now I need to do the same thing in PySpark. I hav… Read more Pyspark Merge Multiple Columns Into A Json Column

How to merge multiple rows into single cell based on id using PySpark? I have a dataframe with ids … Read more How To Merge Multiple Rows Into Single Cell Based On Id And Then Count?

This is part of my python-spark code which parts of it run too slow for my needs. Especially this p… Read more Improve Speed Of Spark App

I have a dataframe named result in pyspark and I want to apply a udf to create a new column as belo… Read more Apply Udf To Multiple Columns And Use Numpy Operations

I am running a PySpark job that calls udfs. I know udfs are bad with memory and slow due to seriali… Read more Error Pythonudfrunner: Python Worker Exited Unexpectedly (crashed)

We distribute our Python app, which uses Spark, together with Python 3.7 interpreter (python.exe wi… Read more Change Pyspark_python On Spark Workers

I have following Dataset drug_name,num_prescriber,total_cost AMBIEN,2,300 BENZTROPINE MESYLATE,1,15… Read more How To Compare Strings Without Case Sensitive In Spark Rdd?