Combine or query two large CSV files?

[timswim78]

At work, I often use Access to query the data in CSV files. Normally, my CSV files are about 500MB, and I simply create a linked table in Access to the CSV files. I then run my queries, and everything is great.

On a current project, I have a large data set that is split across two CSV files (that is how I got it, and the distributor of the data only distributes it that way). Each CSV files is over 1GB. What is the best way to combine these files? Or can I somehow link to both files and write queries that access the data in both CSV files?

At work, I am working on XP Pro, no admin access, and Access is the only tool at my disposal. If need be, I can take the files home on a flash drive to combine them, since they are publicly available data.

 

[WobbleWobble]

In Command prompt, you can go "copy file1.csv + file2.csv combined.csv"

The key thing to watch out for is to make sure file1.csv has new line at the end or else it will merge the last line of file1 and first line of file2.

 

[timswim78]

Originally posted by: WobbleWobble
In Command prompt, you can go "copy file1.csv + file2.csv combined.csv"

The key thing to watch out for is to make sure file1.csv has new line at the end or else it will merge the last line of file1 and first line of file2.

Thanks for the advice. I already tried this, without success. When I tried linking the combined file in Access, I got an error message saying that the file contained no data. The file was 2GB, so I am not sure what happened.

 

[WobbleWobble]

It's a simple file operation, so I can't really see where it went wrong unless it included a second set of header rows or merged the last line of first file didn't have an EOL.

What version of Access? Are you pushing the limits of it?