Python Development

I'm trying the mongodb hadoop integration with spark but can't figure out how to make the j… Read more Add Jar To Pyspark When Using Notebook

I asked the question a while back for python, but now I need to do the same thing in PySpark. I hav… Read more Pyspark Merge Multiple Columns Into A Json Column

I'm trying to learn machine learning with PySpark. I have a dataset that has a couple of String… Read more Pyspark : Keyerror When Converting A Dataframe Column Of String Type To Double

so the problem I have is I have this dataset: and it shows the businesses are doing business in th… Read more Pyspark Sql Compare Records On Each Day And Report The Differences

How to merge multiple rows into single cell based on id using PySpark? I have a dataframe with ids … Read more How To Merge Multiple Rows Into Single Cell Based On Id And Then Count?

This is part of my python-spark code which parts of it run too slow for my needs. Especially this p… Read more Improve Speed Of Spark App

I have a dataframe named result in pyspark and I want to apply a udf to create a new column as belo… Read more Apply Udf To Multiple Columns And Use Numpy Operations

I followed this article to send some data to AWS ES, and I used the jar elasticsearch-hadoop. Here … Read more Write Data From Pyspark To Elasticsearch