Automatically Limit Rows Per File And Create New Files

I am working on a script that will write a massive amount of data to a .csv file. I would like to automatically limit rows per file and create new files.

Solution 1:

A simple approach would be to use a class to keep track of your rows and write to a new file whenever needed (e.g. self.max_row)

For example:

import csv

class MyCSV:
    def __init__(self):
        self.max_row = 10
        self.cur_row = 0
        self.file_number = 0
        self.file_handle = None
        
    def write_row(self, row):
        if self.cur_row >= self.max_row or self.file_handle == None:
            self.cur_row = 0
            self.file_number += 1
            
            if self.file_handle:
                self.file_handle.close()
                
            self.file_handle = open(f'output_{self.file_number:04}.csv', 'w', newline='')
            self.csv_handle = csv.writer(self.file_handle)
            
        self.csv_handle.writerow(row)
        self.cur_row += 1


my_csv = MyCSV()

for row in range(1000):   # create some simulated rows
    output_row = [row, "value1", "value2"]
    my_csv.write_row(output_row)

This would create output filenames of the form output_0001.csv containing 10 rows per file. Obviously you can adjust this as needed.

You could also use a csv.DictWriter() instead and pass a dictionary for each row.