Upload and process CSV file with Python Django

CSV file processing without Database

Hands-on recipe for uploading and processing CSV file with Python Django

Django is a Python-based free and open-source web framework, which follows the model-template-view architectural pattern. It is maintained by the Django Software Foundation. Django's primary goal is to ease the creation of complex, database-driven websites.


With Django, you can take Web applications from concept to launch in a matter of hours. Django takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source.

Django includes dozens of extras you can use to handle common Web development tasks. Django takes care of user authentication, content administration, site maps, RSS feeds, and many more tasks — right out of the box.


In this recipe we will show you how to upload a csv file and then process the content without storing file on server. One approach could be uploading the file, storing it in upload directory and then reading the file.Another approach could be uploading file and reading it directly from post data without storing it in memory and displaying the data. We will work with the later approach here.

Uploading CSV file:
First create HTML form to upload the csv file. Use below code for the same. 

<form action="{% url "myapp:upload_csv" %}" method="POST" enctype="multipart/form-data" class="form-horizontal">
{% csrf_token %}
<div class="form-group">
<label for="name" >File: </label>
<div>
<input type="file" name="csv_file" id="csv_file" required="True" class="form-control">
</div>                   
</div>
<div class="form-group">                   
<div>
<button> <span></span>Upload </button>
</div>
</div>
</form>

Important: Do not forget to include enctype="multipart/form-data"  in form.

Add a URL in URLpatterns. 


url(r'^upload/csv/$', views.upload_csv, name='upload_csv'),

Create a function in views with the name upload_csv .

Process the CSV file:
In view function, get the file from post data and process it. I used below code in my project.


def upload_csv(request):
data = {}
if "GET" == request.method:
return render(request, "myapp/upload_csv.html", data)
# if not GET, then proceed with processing
try:
csv_file = request.FILES["csv_file"]
if not csv_file.name.endswith('.csv'):
messages.error(request,'File is not CSV type')
return HttpResponseRedirect(reverse("myapp:upload_csv"))
#if file is too large, return message
if csv_file.multiple_chunks():
messages.error(request,"Uploaded file is too big (%.2f MB)." % (csv_file.size/(1000*1000),))
return HttpResponseRedirect(reverse("myapp:upload_csv"))

               file_data = csv_file.read().decode("utf-8")          

               lines = file_data.split("\n")
#loop over the lines and save them in db. If error shows up , store as string and then display
for line in lines:                                          
fields = line.split(",")
data_dict = {}
data_dict["name"] = fields[0]
data_dict["start_date_time"] = fields[1]
data_dict["end_date_time"] = fields[2]
data_dict["notes"] = fields[3]
try:
form = EventsForm(data_dict)
if form.is_valid():
form.save()                                  
else:
logging.getLogger("error_logger").error(form.errors.as_json())                                                                                    
except Exception as e:
logging.getLogger("error_logger").error(repr(e))                             
pass

        except Exception as e:
logging.getLogger("error_logger").error("Unable to upload file. "+repr(e))
messages.error(request,"Unable to upload CVS file. "+repr(e))

        return HttpResponseRedirect(reverse("myapp:upload_csv"))

In the above code we are performing below actions:

- If this is a GET request, render the upload csv html file.
- If this is a POST request, proceed.
- First step is to check file name extension. So if the file name is not ending with .csv then this is not the valid file. You may implement you own or additional checks as well.
- Next we check if file is too large. If these tests fail, we return to html form page with appropriate error message. For displaying error/success messages, we are using messages framework. Please import required modules.
- Next we read the file and split the content by new line character.
- Iterate over each line and split the line using comma.
- For following this recipe, assume our csv file have 4 columns of data. We stored the data in a dictionary and then pass the data dictionary to a form.
- If form is valid, we proceed to save the form and hence creating entry in DB.
- If form is not valid, or any other error is thrown, we log the error in log file.

Summary
In this short tutorial, you learn how to process CVS file upload on the fly without storing it on the database step-by-step. As you noticed Django is very powerful tool for handling and processing html forms.


View Our Classes!