Skip to content
This repository was archived by the owner on Sep 14, 2019. It is now read-only.

Skipwhitespace #76

Merged
catherinedevlin merged 3 commits into from
Jul 25, 2016
Merged

Skipwhitespace #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6a80fbd
Remove library that is not needed
vrajmohan Jul 21, 2016
d95be73
Remove unnecessary imports
vrajmohan Jul 21, 2016
9a7349d
Skip spaces after delimiter
vrajmohan Jul 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ invoke==0.10.1
pandas
# intentionally leaving pandas version blank, b/c `pip install` generally
# fails on Linux - must use the package manager version
psycopg2==2.6.1
requests==2.7.0
gunicorn==19.3.0
cryptography>=1.0
Expand Down
8 changes: 4 additions & 4 deletions utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
import logging
import multiprocessing
import os
import tempfile
import threading

import pandas as pd
import sqlalchemy as sa
Expand Down Expand Up @@ -87,11 +85,13 @@ def load_table(filename,
engine = engine or sa.create_engine(config.SQLA_URI)
file = ensure_csv(filename)
# Pass data types to iterator to ensure consistent types across chunks
dtypes = pd.read_csv(file.name, nrows=infer_size).dtypes
dtypes = pd.read_csv(file.name, nrows=infer_size,
skipinitialspace=True).dtypes

@catherinedevlin catherinedevlin Jul 25, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's nice, I hadn't known about skipinitialspace!

chunks = pd.read_csv(file.name,
chunksize=chunk_size,
iterator=True,
dtype=dtypes)
dtype=dtypes,
skipinitialspace=True)
for idx, chunk in enumerate(chunks):
chunk.index += chunk_size * idx
sql_engine = pandasSQL_builder(engine)
Expand Down