Tales from the intern

It's a weird experience introducing yourself as the intern at 27 years old. Begs the question, what the heck you've been doing in your 20s you know?

Well, I travelled a lot and ticked off many things on my bucket list that most only get around to when they're too old (and unfit).

The flip side to being above the intern's average age is that you bring a lot of world experience to the table and are not a flight risk. You're there not because your parents told you it would look good on your cv or to tick off an academic requirement for a university program. But for truly wanting to work on hard problems with likeminded people, and hopefully make good friends along the way (corny but true). You also bring a degree of maturity that is not expected of an intern, but very welcome.

The jist of how the first months went and what I worked on are here and here.

If there is one rule I followed religiously is being able to explain your code line by line. For many that sounds absurd, even archaic. Now I mostly SSH into my VPS through Zed or cmux, and spin up agents in omp, barely reading the code anymore. But as an intern with little SWE experience, it was incredibly beneficial for me to learn how to properly debug on good old VS Code, stepping in and out of functions, placing breakpoints, and actually being able to stand by my code. Trust me, I've seen interns rely on AI coding way too much and look like absolute dorks when questioned by a senior during a PR review. All this happens in an office btw, no remote work, no hiding behind LLMs. You work on your code, sure you can develop it with AI but the AI can't understand it for you. Will the AI be in the room with you when your boss asks you a question about your code?

I digress. It's awesome being the less experienced person in the room, and it's often the most rewarding because your learning curve shoots up exponentially faster than your more senior peers.

Once we had a client demo for a GenAI product I developed. My senior told me he would start it at 1pm, so DEV and UAT environments should obviously remain stable until then. But it's 12pm and I had a user story to finish that required a change in the db. One of the most important things the demo needs are the prompts in the db to send to AI with the RAG-processed document to populate the output in a spreadsheet.

So what happened? I altered a prompt in the db.

Except, I forgot the WHERE clause and altered ALL OF THEM!

There is my ceremonial bad SQL commit. Thank god for point-in-time restores and for my senior, an absolute legend for dealing with it with such steeze.

I had gotten so used to executing SQL queries that I overlooked the most important clause and forgot that changes are irreversible. From then on, I always triple-checked my queries and for more complex queries I used a neat trick: putting BEGIN TRANSACTION; before the SQL query. This showed me the result of the test query without altering the db. When happy with the output, run COMMIT TRANSACTION; Otherwise, ROLLBACK TRANSACTION;

Another fun story: remember the litellm supply chain attack? Guess who tried to install it the day it happened. Yours truly!

I wanted to help one of the AI engineers with her model evals after OpenAI announced the model deprecations. And I forget what exactly happened, but inbetween several meetings I recall trying to install the package. It wasn't until 8pm when I got home that I read Karpathy's tweet. I rushed to the office thinking all the team's secrets were cooked. Thankfully, we need to have IT approve any package before it's installed. What's hilariously freaky is that had that security layer not been there, it still would have failed! I attempted to pip install litellm exactly 1 minute after Pypi quarantined it. Look at the screenshots.

The takeaway lesson from this is to never pip install any version of any new package you want to use. You want to pick a recent version, that is stable and has the best feedback. Not just the newest version.

Anyway, plenty more to say on what I got involved in (eg. vector embeddings, prompt engineering, context rot, multi-concurrency, CI/CD pipelines etc.) but I will leave it here.