Have you considered not using xls as a format to store 1.5GB.
this post was submitted on 22 Jun 2023
25 points (100.0% liked)
Linux
47591 readers
912 users here now
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
founded 5 years ago
MODERATORS
Yes, but it was a request of my boss
Tell them to use a proper database
Good luck with that.
Their boss probably doesn’t care since it opened in excel for them.
Call him special for even entertaining a 1.5GB XLS file as a real option for anything.
That’s totally nuts. Normalize the data and use Access if you have absolutely no other options - or use SQL Server Express if you want to be cheap about it and stay in the MSFT ecosystem.
@meat_popsicle @paf129 @meowthschwitz Or you could take a look at HDF5. if it's good enough for scientists all around the world it should be good for him too.
Microsoft never really meant for XLS files to get that big. Can't you use a "saner" approach, like CSV?
If it is data processing, tell them to use Octave or even better Python + spyder + pandas & matplotlib/seaborn.
Much easier processing, much less time wasted with opening more flexible, and more beautiful plots.
Yeah, you can make it work by finding one of your it guys and dumping it in a real database. Spreadsheet software is NOT meant to handle data like that.
Can you split it into a few smaller files? That is honestly too large for any spreadsheet.
Dang! How does it take to open that file? Are there formulas and macros in there?
As the others said, a spreadsheet isn't a database. And you now experience why. Tell your boss it's not going to work anymore.
Damn, a 1.5GB Excel file!? That sounds like a nightmare to work with. And here I thought a 200MB Excel file acting as a store’s entire yearly accounting ledger (where I used to work) was bad!
At this point, you need a proper database and some reporting scripts / software. If that one file somehow gets corrupted, it’s all over.
This is why exel is not a database. I'd say export to a reasonable format on Windows (like CSV, but even that might be too much) and then do the work on
For real, what are you using this one for? If I take the compression ratio into account (times 10), subtract meta data (minus a third) and consider the character encoding (two bytes per character) I still get around 5000000000 characters. That's all numbers? You wouldn't store binary content in there, would you?
Anyway, take a decent pc with at least twice the memory the file will need uncompressed. Kill all other applications. Open up your file and split it up. Make a separate file for each worksheet. If there are more than 100000 rows, split them in separate files too. Don't embed any binary content.
And for god's sake: use a different tool. That very much depends on what you are trying to do. What is it?
It is just numbers, but my boss wants it in excel because he can't work another file :(
You could use office in wine , maybe wps office will work with it.
Hi! Try to open it with Apache Spark (difficult to learn at the beginning) or Pandas in Python (more friendly option). Use batch reading and dump the information in a Database (PostgreSQL) or a plain text format file (csv).
Excuse me, I have a problem transporting this concrete bridge segment on my VW beatle. It worked fine with my Honda Accord (just made it a bit slow) but the beatle just gets crushed under it. Anyone can tell me how to make it work?
Edit: car type to make more appropriate comparison