Alright, so today I’m gonna walk you through this thing I was messing with called “jc encarnacion”. Sounds fancy, right? Nah, it’s just a name I slapped on a little project I was tinkering with.
It all started when I was trying to figure out a better way to, like, organize some data I was pulling from different places. I was tired of spreadsheets and clunky databases. I wanted something cleaner, something I could actually use without wanting to throw my laptop out the window.
So, first thing I did was fire up my code editor. I’m a Python guy, so that’s where I started. I figured, “Okay, let’s try to wrangle this data into something manageable.” I started by defining some classes – nothing too complicated, just basic stuff to represent the kind of data I was dealing with. Think of it like making little containers for all the different pieces of information.
Then came the fun part – actually getting the data. I had to write a bunch of scripts to scrape info from a couple of websites and pull data from an old database. Let me tell you, dealing with web scraping can be a real pain. Websites change all the time, so you have to keep updating your scripts to keep them working. It’s like playing whack-a-mole, but with code.
Once I had the data, I needed to clean it up. This is where things got messy. There were missing values, inconsistent formats, and all sorts of other garbage. I spent a solid chunk of time writing functions to handle all the different cleaning tasks. It was tedious, but necessary. You can’t do anything useful with data that’s all messed up.
After cleaning, I started thinking about how to store the data. I considered a few options, like a NoSQL database or just plain old JSON files. But I wanted something that was easy to query and analyze, so I ended up using SQLite. It’s simple, lightweight, and perfect for a small project like this.
Next step was writing the code to actually load the data into the SQLite database. I created a table schema, and then wrote a script to iterate over the cleaned data and insert it into the table. It took a couple of tries to get it right, but eventually, I had all my data neatly stored in the database.
Now that the data was in a usable format, I could finally start doing something with it. I wrote some simple queries to extract different pieces of information and generate some basic reports. I even created a little web interface using Flask so I could view the data in my browser. It wasn’t pretty, but it worked.
The whole process took me a few days, but it was totally worth it. I learned a lot about data wrangling, web scraping, and database management. And more importantly, I now have a much better way to organize and analyze my data. It’s not perfect, but it’s a hell of a lot better than spreadsheets.
The final step? I decided to just save everything to Github, so I can share to other people.